solution: docs for access logging

2021-07-19 23:15:31 -04:00
parent 3a12efd0a0
commit e8fc025966
11 changed files with 141 additions and 25 deletions
--- a/docs/01-architecture-intro.adoc
+++ b/docs/01-architecture-intro.adoc
@@ -29,7 +29,7 @@ And for a request:
 - Is it for concrete data (_block #100_) or the latest (_get balance_)?
 - Is result a static value (_just block_) or may vary depending on network and node (_latest nonce_)?
 - Does it need to be repeated over multiple nodes (_broadcast transaction_)?
- Request can also specify which subset of nodes should be able to execute the request by selecting node _Labels_ (see "link:08-quorum-and-selectors.adoc[Quorum and Selectors]")
+- Request can also specify which subset of nodes should be able to execute the request by selecting node _Labels_ (see "link:09-quorum-and-selectors.adoc[Quorum and Selectors]")

 Based on these factors, Dshackle executes the request on a most optimal node.
 For most of the simple requests, it just gets a node that is synchronized to the point that must have a response for that particular request.
@@ -43,7 +43,7 @@ It provides:
 - Routes only realy/alive upstreams, i.e., synchronized and with enough peers
 - Load Balancing
 - Request Retry on upstream errors
- Local Caching (memory and Redis, see link:09-caching.adoc[Caching])
+- Local Caching (memory and Redis, see link:10-caching.adoc[Caching])
 - Broadcasting and Quorum for requests

 === gRPC protocol
--- a/docs/02-quick-start.adoc
+++ b/docs/02-quick-start.adoc
@@ -174,4 +174,4 @@ grpcurl -import-path ./proto/ -proto blockchain.proto -d '{"asset": {"chain": "1
 ...
 ----

-See other enhanced methods in the link:06-methods.adoc[Documentation for Enhanced Methods]
+See other enhanced methods in the link:07-methods.adoc[Documentation for Enhanced Methods]
--- a/docs/03-server-config.adoc
+++ b/docs/03-server-config.adoc
@@ -48,7 +48,8 @@ a| `port: 12449`

 a| `tls`
 |
-| TLS configuration for gRPC. See link:07-authentication.adoc[Authentication] for details
+| TLS configuration for gRPC.
+See link:08-authentication.adoc[Authentication] for details

 a| `proxy`
 |
@@ -121,7 +122,8 @@ a| `enabled: true`

 a| `tls`
 |
-| TLS configuration for proxy. See link:07-authentication.adoc[Authentication] for details
+| TLS configuration for proxy.
+See link:08-authentication.adoc[Authentication] for details

 a| `routes`
 |
--- a/docs/06-monitoring.adoc
+++ b/docs/06-monitoring.adoc
@@ -0,0 +1,74 @@
+= Logging & Monitoring
+
+== Access / Request Log
+
+Dshackle can log all requests to a file in JSON format.
+Or https://jsonlines.org/[JSON Lines] to be more precise, i.e., a test file where each line is a JSON.
+
+NOTE: By default, the access log is disabled.
+
+To enable access log add following configuration:
+
+[source,yaml]
+----
+accessLog:
+  enabled: true
+  filename: /var/log/dshackle/access_log.jsonl
+----
+
+`filename` is optional, and the default value is `access_log.jsonl` (i.e., in the current directory).
+
+Since a single request may contain multiple replies (ex., a batch call, or subscribe to the head blocks) the Dshackle logging is based on replies.
+The access log file contains details per each response send from the server, and each of them refers to original request details.
+
+The access log contains the JSON lines similar to:
+
+[source,json]
+----
+{
+  "version":"accesslog/v1beta",
+  "ts":"2021-07-20T01:53:33.174645Z",
+  "id":"578d83db-cf53-4ef8-b73e-3f1cc0a67e96",
+  "method":"NativeCall",
+  "blockchain":"ETHEREUM",
+  "total":2,
+  "index":0,
+  "succeed":true,
+  "request":{
+    "id":"513b9b49-b472-4c83-b4b7-58dd2aabe9f6",
+    "start":"2021-07-20T01:53:33.086946Z",
+    "remote":{
+      "ips":["127.0.0.1", "10.0.5.102", "172.217.8.78"],
+      "ip":"172.217.8.78",
+      "userAgent":"grpc-node-js/1.1.8"
+    }
+  },
+  "nativeCall":{
+     "method":"eth_blockNumber",
+     "id":2,
+     "payloadSizeBytes":2
+  }
+}
+----
+
+.Where:
+- `ts` timestamp of the reply
+- `id` uniq id of the reply
+- `method` Dshackle method which was called (i.e., not a Blockchain API method, see `nativeCall` details)
+- `blockchain` blockchain code
+- `total` how many requests in the batch (available only for a `NativeCall` call)
+- `index` current index (i.e. count) of the reply to the original request
+- `succeed` if call succeeded, in terms of Blockchain API
+- `request` original request details
+** `id` uniq id of the request; all replied to the same request have same id
+** `start` when request was received
+** `remote` remote details
+*** `ips` list of all recognized IPs (including headers such as `X-Real-IP` and `X-Forwarded-For`)
+*** `ip` a single ip, that likely represent a real IP of the remote
+*** `userAgent` user agent
+- `nativeCall` details of the individual Native Call request
+** `method` method name terms of Blockchain API
+** `id` request id provided in the original request
+** `payloadSizeBytes` size of the original _individual_ request (for JSON RPC it's size of the `params` value)
+
+
--- a/docs/07-methods.adoc
+++ b/docs/07-methods.adoc
@@ -170,4 +170,4 @@ message TxStatus {

 === gRPC Client Libraries

-See link:10-client-libraries.adoc[Client Libraries] documentation.
+See link:11-client-libraries.adoc[Client Libraries] documentation.
--- a/docs/08-authentication.adoc
+++ b/docs/08-authentication.adoc
--- a/docs/09-quorum-and-selectors.adoc
+++ b/docs/09-quorum-and-selectors.adoc
--- a/docs/10-caching.adoc
+++ b/docs/10-caching.adoc
--- a/docs/11-client-libraries.adoc
+++ b/docs/11-client-libraries.adoc
--- a/docs/README.adoc
+++ b/docs/README.adoc
@@ -38,11 +38,12 @@ Main goals:
 . link:03-server-config.adoc[Server Configuration]
 . link:04-upstream-config.adoc[Upstreams Configuration]
 . link:05-start.adoc[How to launch a server]
-. link:06-methods.adoc[API methods]
-. link:07-authentication.adoc[Authentication]
-. link:08-quorum-and-selectors.adoc[Quorum and Selectors]
-. link:09-caching.adoc[Caching]
-. link:10-client-libraries.adoc[Client Libraries]
+. link:06-monitoring.adoc[Logging & Monitoring]
+. link:07-methods.adoc[API methods]
+. link:08-authentication.adoc[Authentication]
+. link:09-quorum-and-selectors.adoc[Quorum and Selectors]
+. link:10-caching.adoc[Caching]
+. link:11-client-libraries.adoc[Client Libraries]

 == Reference

--- a/docs/reference-configuration.adoc
+++ b/docs/reference-configuration.adoc
@@ -58,6 +58,10 @@ tokens:
    type: ERC-20
    address: 0xdac17f958d2ee523a2206206994597c13d831ec7

+accessLog:
+  enabled: true
+  filename: /var/log/dshackle/access_log.jsonl
+
 cluster:
  defaults:
    - chains:
@@ -142,20 +146,29 @@ cluster:

 | `tls`
 |
-| Setup TLS configuration for the gRPC server. See <<tls>> section
+| Setup TLS configuration for the gRPC server.
+See <<tls>> section

 | `proxy`
 |
-| Setup HTTP proxy that emulates all standard JSON RPC requests. See <<proxy>> section
+| Setup HTTP proxy that emulates all standard JSON RPC requests.
+See <<proxy>> section
+
+| `accessLog`
+|
+| Configure access logging.
+See <<accessLog>> section
+

 | `tokens`
 |
-| Configure tokens for tracking balance. See <<tokens>> section
-
+| Configure tokens for tracking balance.
+See <<tokens>> section

 | `cache`
 |
-| Caching configuration. See <<cache>> section.
+| Caching configuration.
+See <<cache>> section.

 | `cluster`
 |
@@ -265,6 +278,31 @@ a| Routing paths for Proxy. The proxy will handle requests as `https://${HOST}:$

 |===

+[#accessLog]
+== Access Log config
+
+[source,yaml]
+----
+accessLog:
+  enabled: true
+  filename: /var/log/dshackle/access_log.jsonl
+----
+
+.Access Log config
+[cols="2a,3a,7"]
+|===
+| Option | Default | Description
+
+| `enabled`
+| `false`
+| Enable/Disable Access logging
+
+| `filename`
+| `access_log.jsonl`
+| Path to the access log file
+
+|===
+
 [#tokens]
 == Tokens config

@@ -284,7 +322,7 @@ tokens:
 ----

 Tokens config enables tracking of a balance amount in the configured tokens.
-After making the configuration above you can request balance (`GetBalance`), or subscribe to balance changes (`SubscribeBalance`), using link:06-methods.adoc[enhanced protocol]
+After making the configuration above you can request balance (`GetBalance`), or subscribe to balance changes (`SubscribeBalance`), using link:07-methods.adoc[enhanced protocol]

 .Token config
 [cols="2a,7"]
@@ -446,7 +484,7 @@ Accepted types: `bitcoin`, `bitcoin-testnet`, `ethereum`, `ethereum-classic`, `k
 | no
 | Key-Value pairs that are assigned to the upstream.
 Used to select an upstream per-request.
-See link:08-quorum-and-selectors.adoc[Quorum and Selectors]
+See link:09-quorum-and-selectors.adoc[Quorum and Selectors]

 | `methods`
 | no
@@ -538,9 +576,10 @@ It's more effective, easier to secure connection, and allows to build a distribu
 | Address to connect to

 | `tls`
-a| TLC configuration for the connection. In general it's an optional configuration, but it's strongly recommended. Also
-HTTP2 + gRPC is designed to be used with TLS, and some of the related software is unable to use it without TLS. +
-  See link:07-authentication.adoc[Authentication] docs and <<tls>>.
+a| TLC configuration for the connection.
+In general it's an optional configuration, but it's strongly recommended.
+Also HTTP2 + gRPC is designed to be used with TLS, and some of the related software is unable to use it without TLS. +
+See link:08-authentication.adoc[Authentication] docs and <<tls>>.

 | `tls.ca`
 | Path to x509 certificate to verify remote server