:imagesdir: assets

= Logging & Monitoring

== Access / Request Log

Dshackle can log all requests to a file in JSON format.
Or https://jsonlines.org/[JSON Lines] to be more precise, i.e., a test file where each line is a JSON.

NOTE: By default, the access log is disabled.

To enable access log add following configuration:

[source,yaml]
----
accessLog:
  enabled: true
  filename: /var/log/dshackle/access_log.jsonl
----

`filename` is optional, and the default value is `access_log.jsonl` (i.e., in the current directory).

Since a single request may contain multiple replies (ex., a batch call, or subscribe to the head blocks) the Dshackle logging is based on replies.
The access log file contains details per each response send from the server, and each of them refers to original request details.

The access log contains the JSON lines similar to:

[source,json]
----
{
  "version":"accesslog/v1beta",
  "ts":"2021-07-20T01:53:33.174645Z",
  "id":"578d83db-cf53-4ef8-b73e-3f1cc0a67e96",
  "method":"NativeCall",
  "channel":"GRPC",
  "blockchain":"ETHEREUM",
  "total":2,
  "index":0,
  "succeed":true,
  "request":{
    "id":"513b9b49-b472-4c83-b4b7-58dd2aabe9f6",
    "start":"2021-07-20T01:53:33.086946Z",
    "remote":{
      "ips":["127.0.0.1", "10.0.5.102", "172.217.8.78"],
      "ip":"172.217.8.78",
      "userAgent":"grpc-node-js/1.1.8"
    }
  },
  "nativeCall":{
     "method":"eth_blockNumber",
     "id":2,
     "payloadSizeBytes":2
  }
}
----

.Where:
- `ts` timestamp of the reply
- `id` uniq id of the reply
- `method` Dshackle method which was called (i.e., not a Blockchain API method, see `nativeCall` details)
- `blockchain` blockchain code
- `channel` access channel (`GRPC` for native Dshackle calls, `JSONRPC` for JSON RPC HTTP Proxy)
- `total` how many requests in the batch (available only for a `NativeCall` call)
- `index` current index (i.e. count) of the reply to the original request
- `succeed` if call succeeded, in terms of Blockchain API
- `request` original request details
** `id` uniq id of the request; all replied to the same request have same id
** `start` when request was received
** `remote` remote details
*** `ips` list of all recognized IPs (including headers such as `X-Real-IP` and `X-Forwarded-For`)
*** `ip` a single ip, that likely represent a real IP of the remote
*** `userAgent` user agent
- `nativeCall` details of the individual Native Call request
** `method` method name terms of Blockchain API
** `id` request id provided in the original request
** `payloadSizeBytes` size of the original _individual_ request (for JSON RPC it's size of the `params` value)

== Prometheus Metrics

By default, Dshackle provides Prometheus metrics on `http://127.0.0.1:8081/metrics`.

To configure the metrics use:

[source,yaml]
----
monitoring:
  enabled: true
  jvm: false
  extended: false
  prometheus:
    enabled: true
    bind: 192.168.0.1
    port: 8000
    path: /status/prometheus
----

Where `jvm` options enabled monitoring of the JVM internals, such as memory, GC, threads, etc.
And `extended` enables additional metrics for query selectors, etc.

== Grafana Dashboard

Simple Grafana dashboard available link:../dashboard/dshackle.json[here]

image::dshackle-dashboard.png[alt="",width=80%,align="center"]

This dashboard contains:

- Upstreams Availability

- Upstreams Lag

- JSON RPC total request / failed requests

- GRPC total request / failed requests

- JSON RPC Response time

- Upstreams Errors

- JSON RPC upstream conn seconds 50,75,90,99 percentiles

== Health Checks

Dshackle provides a http endpoint to check status of the servers.
This check is compatible with https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes[Kubernetes Liveness and Readiness Probes].

By default, it's disabled, and you have to set up which blockchain are required to be available to consider Dshackle alive.

.Example config:
[source,yaml]
----
health:
  port: 8082 # <1>
  host: 127.0.0.1 # <2>
  path: /health # <3>
  blockchains: # <4>
    - chain: ethereum # <5>
      min-available: 2 # <6>
    - chain: bitcoin
      min-available: 1
----

<1> (optional) port to bind the Health server.
Default: `8082`
<2> (optional) host to bind the Health server.
Default: `127.0.0.1`
<3> (optional) path on the server.
Default: `/health`.
I.e., `http://127.0.0.1:8082/health` with default config
<4> list of blockchain to check availability
<5> a Blockchain to check
<6> minimum available (i.e., fully synced) Upstreams for that blockchain

With the config above the server is considered healthy if:

- Dshackle has connected to at least two valid Ethereum upstreams
- **and** at least one valid Bitcoin upstream.

When the server is healthy is responds with `OK` and 200 as HTTP Status Code.
When any of the checks failed, it responds with a short description and 503 as HTTP Status Code.

Example of a response for an unhealthy server that doesn't have enough upstreams for a Ethereum Classic Blockchain.

.GET http://127.0.0.1:8082/health
----
ETHEREUM_CLASSIC UNAVAILABLE
----

Optionally, the server can be called with `?detailed` query, which provides a more detailed response:

.GET http://127.0.0.1:8082/health?detailed
----
ETHEREUM_CLASSIC UNAVAILABLE
BITCOIN AVAILABLE
  local-btc-1 OK with lag=0
ETHEREUM AVAILABLE
  local-eth-1 OK with lag=0
  local-eth-2 OK with lag=0
----