== Upstreams Configuration

Dshackle can connect to multiple independent APIs ("upstreams") and provides an unified API on top of it.

Supported upstream protocols:

- JSON RPC
- Websockets
- gRPC (i.e. can connect to another Dshackle)

Those protocols can be configures with additional security, TLS and authentication.

=== Notes on upstream configuration

==== Ethereum

- Most of Ethereum nodes support WebSocket connection, in addition to the JSON RPC.
If it's available on your node, it's suggested to configure both JSON RPC and WebSocket connection

==== Ethereum PoS

- The support for PoS Ethereum isn't fully implemented yet. However, we support it in a simplified way.
As we can no longer rely on the difficulty parameter of a block for fork choice algorithm we implement upstream rating.
In short terms, we just consider the upstream with the highest rating to be always correct when reporting its head.
Unless it's down then we will fallback on the upstream with the second highest rating, etc.

==== Bitcoin

- Bitcoind needs to be configured to index/track addresses that you're going to request.
Make sure you configure it to index your addresses with `importaddress`.
If you request balance for an address that is not indexed then it returns 0 balance.
- To track all transactions you need to setup index for transactions, which is disabled by default.
Run it with `-reindex` option, or set `txindex=1` in the config.

=== Example Configuration

.upstreams.yaml
[source,yaml]
----
version: v1

cluster:
  defaults:
  - chains:
      - ethereum
    options:
      min-peers: 10
  - chains:
      - kovan
    options:
      min-peers: 2
  upstreams:
    - id: us-nodes
      node-id: 1
      chain: auto
      method-groups:
       enabled:
         - trace
      connection:
        grpc:
          host: 35.226.252.117
          port: 443
          tls:
            ca: ca.crt
            certificate: client.crt
            key: client.p8.key
    - id: infura-eth
      node-id: 2
      chain: ethereum
      role: fallback
      labels:
        provider: infura
      options:
        disable-validation: true
      connection:
        ethereum-pos:
            execution:
              rpc:
                url: "https://mainnet.infura.io/v3/${INFURA_USER}"
                basic-auth:
                  username: ${INFURA_USER}
                  password: ${INFURA_PASSWD}
              ws:
                url: "wss://mainnet.infura.io/ws/v3/${INFURA_USER}"
                basic-auth:
                  username: ${INFURA_USER}
                  password: ${INFURA_PASSWD}
    - id: ethereum-pos
      node-id: 3
      chain: ropsten
      connection:
        ethereum-pos:
            execution:
              rpc:
                url: ${ROPSTEN_NODE_RPC_URL}
              ws:
                url: ${ROPSTEN_NODE_WS_URL}
----

There are two main segments for upstreams configuration:

- _upstreams_ - a list of API to connect to, with all configuration specific for upstream and chain
- and _default options_ as common configuration options applied to all nodes in that group

In the example above we have:

- default configuration for _Ethereum Mainnet_ which accepts upstream as valid when it not in fast synchronization mode and has at least 10 peers.For _Kovan Testnet_ nodes the requirements are much relieved
- as upstreams it has 2 configurations
* balancer connects to another Dshackle/another machine by using gRPC protocol
** accepts (i.e. proxies) any blockchain available on that remote
** verifies TLS certificate of the server
** uses client certificate for authentication, i.e. remote server is accepting only clients authenticated by a certificate
* connects to Infura provided _Ethereum Mainnet_
** as a _fallback_ upstream, which means that it's used only if `us-nodes` fails
** configuration is using placeholders for `${INFURA_USER}` and `${INFURA_PASSWD}` which will be replaced with corresponding environment variables values
** uses Basic Authentication to authenticate requests on Infura
** label `[provider: infura]` is set for that particular upstream, which can be selected during a request.For example for some requests you may want to use nodes with that label only, i.e. _"send that tx to infura nodes only"_, or _"read only from archive node, with label [archive: true]"_
** upstream validation (peers, sync status, etc) is disabled for that particular upstream

=== Nodes
`[node-id: 1]` is numeric node identifier defined in a range [1..255] and used to forward
`eth_getFilterChanges` request to the node where one of `eth_newFilter`, `eth_newBlockFilter` or `eth_newPendingTransactionFilter` methods was executed (because filter is s stateful method).

*It's kindly recommended* to strictly associate _node-id_ parameter with a physical node and keep it during any configuration changes

=== Supported chains

Currently, dshackle supports next chains (should be used as chain names in config):

- ethereum (eth)
- ethereum-classic (etc)
- polygon (matic)
- polygon-mumbai
- arbitrum (arb)
- arbitrum-testnet
- arbitrum-nova
- optimism
- optimism-testnet
- binance (bsc, bnb-smart-chain)
- zksync
- zksync-testnet
- polygon-zkevm
- polygon-zkevm-testnet
- morden
- kovan (kovan-testnet)
- goerli (goerli-testnet)
- rinkeby (rinkeby-testnet)
- ropsten (ropsten-testnet)
- bitcoin (bitcoin-testnet)

=== Roles and Fallback upstream

By default, the Dshackle connects to each upstream in a Round-Robin basis, i.e. sequentially one by one.
If you need more gradual control over the order of which upstream is used and when you can assign following roles:

- `primary` (default role if nothing specified)
- `secondary`
- `fallback`

Where `primary` and `secondary` are considered here a _standard_ upstreams, and `fallback` is used on failure of standard upstreams.
I.e. the Dshackle always starts with making requests to standard upstreams.
If all of them failed, if responses are inconsistent (ex. for `eth_getTransactionCount`), or when it needs to broadcast to a wider network (`sendrawtransaction`), then upstreams with role `fallback` cames to use.

The internal request order is (goes to next only if all upstreams on current step a not available or failed):

1. tries with primary upstreams
2. tries with secondary upstream
3. ... delay (100ms at first, increased each iteration)
4. tries with primary upstreams
5. tries with secondary upstream
6. tries with fallback upstreams
7. ... go to step 3

Steps 3-6 are repeated until a valid response received, or a timeout for the original request is reached.

In general:
- you set role `secondary` for upstream in another cluster/datacenter - you set role `fallback` for an external upstream which may be provided by a third party, and you want to use it as a last resort

=== Configuration options

Options (default or as part of upstream config):

[cols="2,1,5a"]
|===
| Option | Default | Description

| `disable-validation` | false | if `true` then Dshackle will not try to verify status of the upstream (could be useful for a trusted cloud
provider such as Infura, but disabling it is not recommended for a normal node)
| `min-peers` | 3 | specify minimum amount of connected peers, Dshackle will not use upstream with less than specified number
| `timeout` | 60 | timeout in seconds after which request to the upstream will be discarded (and may be retried on an another upstream)
| `balance` | `true` for ethereum, `false` for bitcoin | specify if this node should be used to fetch balance for an address
|===

=== Connection type

Dshackle currently supports

- `rpc` a standard Ethereum JSON RPC
- `ws` websocket connection (supposed to be used in addition to `rpc` connection)
- `grpc` connects to another Dshackle instance

==== Connection mixture modes
In case of rpc and ws connection we can specify different modes of works together:

|===
|Type |Description

|WS_ONLY
|Default mode in case WS endpoint specified. In this mode WS connection is used for all requests and subscriptions.

|RPC_ONLY
|Default in case WS endpoint not specified. In this mode RPC connection is used for all requests, subscriptions doesn't work, head subscription works through scheduled RPC head request.

|RPC_REQUESTS_WITH_MIXED_HEAD
|All requests are sent through RPC connection, eth_subscribe is sent through WS connection, head subscription works through scheduled RPC head request mixed with WS subscription.

|RPC_REQUESTS_WITH_WS_HEAD
|All requests are sent through RPC connection, all subscriptions works through WS connection.
|===

You can specify this modes through `connector-mode` parameter in connection config.

=== Bitcoin Methods

.By default an ethereum upstream allows call to the following JSON RPC methods:
- `getbestblockhash`
- `getblock`
- `getblocknumber`
- `getblockcount`
- `gettransaction`
- `getrawtransaction`
- `gettxout`
- `getreceivedbyaddress`
- `listunspent`
- `sendrawtransaction`

.Plus following methods are answered directly by Dshackle
- `getmemorypool`
- `getconnectioncount`
- `getnetworkinfo`

=== Ethereum Methods

.By default, an ethereum upstream allows calls to the following JSON RPC methods:
- `eth_gasPrice`
- `eth_call`
- `eth_estimateGas`
- `eth_getBlockTransactionCountByHash`
- `eth_getUncleCountByBlockHash`
- `eth_getBlockByHash`
- `eth_getTransactionByHash`
- `eth_getTransactionByBlockHashAndIndex`
- `eth_getStorageAt`
- `eth_getCode`
- `eth_getUncleByBlockHashAndIndex`
- `eth_getTransactionCount`
- `eth_blockNumber`
- `eth_getBalance`
- `eth_sendRawTransaction`
- `eth_getBlockTransactionCountByNumber`
- `eth_getUncleCountByBlockNumber`
- `eth_getBlockByNumber`
- `eth_getTransactionByBlockNumberAndIndex`
- `eth_getTransactionReceipt`
- `eth_getUncleByBlockNumberAndIndex`
- `eth_feeHistory`
- `eth_getLogs`
- `eth_getFilterChanges`
- `eth_newFilter`
- `eth_newBlockFilter`
- `eth_newPendingTransactionFilter`

.Plus following methods are answered directly by Dshackle
- `net_version`
- `net_peerCount`
- `net_listening`
- `web3_clientVersion`
- `eth_protocolVersion`
- `eth_syncing`
- `eth_coinbase`
- `eth_mining`
- `eth_hashrate`
- `eth_accounts`

It's possible to enable additional methods that are available on upstream, or disable an existing method.
For that purpose there is `methods` configuration:

[source, yaml]
----
upstreams:
  - id: my-node
    chain: ethereum
    labels:
      archive: true
    methods:
      enabled:
        - name: trace_transaction
      disabled:
        - name: eth_getBlockByNumber
----

Such configuration option allows executing method `trace_transaction` and also disables `eth_getBlockByNumber` on that particular upstream.
If a client requests to execute method `trace_transaction` then it will be scheduled to that upstream (or any upstream with such method enabled).

NOTE: It's especially useful when used together with upstream labels.If an archive upstream has label `archive: true` it's possible to specify that the client wants to execute method `trace_transaction` only on an archive node(s), which has complete historical data for tracing.

==== eth_subscribe ethereum methods
It is possible to control ethereum subscription types like regular methods. For now there are only one option - newPendingTransactions - you need to allow this method if you want to activate this type of subscription for Dshackle upstreams (disabled by default)

=== Static Methods

You can overwrite existing methods or add new ones using a static response:

[source,yaml]
----
upstreams:
  - id: my-node
    chain: ethereum
    methods:
      enabled:
        - name: net_version
          static: "\"100000\""
        - name: eth_chainId
          static: "0x186a0"
        - name: eth_custom_array
          static: '["custom_array_response"]'
        - name: eth_custom_bool
          static: "false"
----

=== Authentication

==== TLS

All connection types can use TLS secured connection, with optional client certificate authentication:

- `ca` path to certificate required from remote server
- optional `certificate` and `key` for client authentication.

NOTE: Please note that `key` must be encoded with _PKCS 8_

==== Basic Authentication

For JSON RPC and Websockets a Basic Authentication can be used:

- `username` - username
- `password` - password

=== Chains specific configuration
We can use chain settings to specify chain specific behavior, for example rules for dshackle to work with upstream statuses

.chains.yaml
[source,yaml]
----
chain-settings:
  default:
    lags:
      syncing: 6
      lagging: 1
  chains:
    - id: eth
      lags:
        syncing: 6
        lagging: 1
    - id: polygon
      lags:
        syncing: 20
        lagging: 10
----
Options
[cols="2,5a"]
|===
| Option | Description

| `lags.syncing` | the size of the lag after which the upstream is determined to be syncing
| `lags.lagging` | the size of the lag after which the upstream is determined to be lagging
|===
