== Upstreams Configuration Dshackle can connect to multiple independent APIs ("upstreams") and provides an unified API on top of it. Supported upstream protocols: - JSON RPC - Websockets - gRPC (i.e. can connect to another Dshackle) Those protocols can be configures with additional security, TLS and authentication. === Notes on upstream configuration ==== Ethereum - Most of Ethereum nodes support WebSocket connection, in addition to the JSON RPC. If it's available on your node, it's suggested to configure both JSON RPC and WebSocket connection ==== Bitcoin - Bitcoind needs to be configured to index/track addresses that you're going to request. Make sure you configure it to index your addresses with `importaddress`. If you request balance for an address that is not indexed then it returns 0 balance. - To track all transactions you need to setup index for transactions, which is disabled by default. Run it with `-reindex` option, or set `txindex=1` in the config. === Example Configuration .upstreams.yaml [source,yaml] ---- version: v1 cluster: defaults: - chains: - ethereum options: min-peers: 10 - chains: - kovan options: min-peers: 2 upstreams: - id: us-nodes chain: auto connection: grpc: host: 35.226.252.117 port: 443 tls: ca: ca.crt certificate: client.crt key: client.p8.key - id: infura-eth chain: ethereum role: fallback labels: provider: infura options: disable-validation: true connection: ethereum: rpc: url: "https://mainnet.infura.io/v3/${INFURA_USER}" basic-auth: username: ${INFURA_USER} password: ${INFURA_PASSWD} ws: url: "wss://mainnet.infura.io/ws/v3/${INFURA_USER}" basic-auth: username: ${INFURA_USER} password: ${INFURA_PASSWD} ---- There're two main segments for upstreams configuration: - _upstreams_ - a list of API to connect to, with all configuration specific for upstream and chain - and _default options_ as common configuration options applied to all nodes in that group In the example above we have: - default configuration for _Ethereum Mainnet_ which accepts upstream as valid when it not in fast synchronization mode and has at least 10 peers.For _Kovan Testnet_ nodes the requirements are much relieved - as upstreams it has 2 configurations * balancer connects to another Dshackle/another machine by using gRPC protocol ** accepts (i.e. proxies) any blockchain available on that remote ** verifies TLS certificate of the server ** uses client certificate for authentication, i.e. remote server is accepting only clients authenticated by a certificate * connects to Infura provided _Ethereum Mainnet_ ** as a _fallback_ upstream, which means that it's used only if `us-nodes` fails ** configuration is using placeholders for `${INFURA_USER}` and `${INFURA_PASSWD}` which will be replaced with corresponding environment variables values ** uses Basic Authentication to authenticate requests on Infura ** label `[provider: infura]` is set for that particular upstream, which can be selected during a request.For example for some requests you may want to use nodes with that label only, i.e. _"send that tx to infura nodes only"_, or _"read only from archive node, with label [archive: true]"_ ** upstream validation (peers, sync status, etc) is disabled for that particular upstream === Fallback upstream By default, the Dshackle connects to each upstream in a Round-Robin basis, i.e. sequentially one by one. But if an upstream have `role: fallback` then it's used only in additional to other (_standard_) upstreams when their responses are not enough to finalize the request. Dshackle always starts with making requests to standard upstreams. If all of them failed, if responses are inconsistent (ex. for `eth_getTransactionCount`), or when it needs to broadcast to wider networks (`sendrawtransaction`), then upstreams with role `fallback` are also used. The internal request order is: 1. connect to each standard upstream 2. delay 3. try again to connect to standard upstreams 4. try to connect to fallback upstreams Steps 2-4 are repeated until a valid response received, or timeout for the original request is reached. In general, you set role `fallback` only to external nodes provided by a third party, when you want to use it as a last resort. === Configuration options Options (default or as part of upstream config): [cols="2,1,5a"] |=== | Option | Default | Description | `disable-validation` | false | if `true` then Dshackle will not try to verify status of the upstream (could be useful for a trusted cloud provider such as Infura, but disabling it is not recommended for a normal node) | `min-peers` | 3 | specify minimum amount of connected peers, Dshackle will not use upstream with less than specified number | `timeout` | 60 | timeout in seconds after which request to the upstream will be discarded (and may be retried on an another upstream) | `balance` | `true` for ethereum, `false` for bitcoin | specify if this node should be used to fetch balance for an address |=== === Connection type Dshackle currently supports - `rpc` a standard Ethereum JSON RPC - `ws` websocket connection (supposed to be used in addition to `rpc` connection) - `grpc` connects to another Dshackle instance === Bitcoin Methods .By default an ethereum upstream allows call to the following JSON RPC methods: - `getbestblockhash` - `getblock` - `getblocknumber` - `getblockcount` - `gettransaction` - `getrawtransaction` - `gettxout` - `getreceivedbyaddress` - `listunspent` - `sendrawtransaction` .Plus following methods are answered directly by Dshackle - `getmemorypool` - `getconnectioncount` - `getnetworkinfo` === Ethereum Methods .By default, an ethereum upstream allows calls to the following JSON RPC methods: - `eth_gasPrice` - `eth_call` - `eth_estimateGas` - `eth_getBlockTransactionCountByHash` - `eth_getUncleCountByBlockHash` - `eth_getBlockByHash` - `eth_getTransactionByHash` - `eth_getTransactionByBlockHashAndIndex` - `eth_getStorageAt` - `eth_getCode` - `eth_getUncleByBlockHashAndIndex` - `eth_getTransactionCount` - `eth_blockNumber` - `eth_getBalance` - `eth_sendRawTransaction` - `eth_getBlockTransactionCountByNumber` - `eth_getUncleCountByBlockNumber` - `eth_getBlockByNumber` - `eth_getTransactionByBlockNumberAndIndex` - `eth_getTransactionReceipt` - `eth_getUncleByBlockNumberAndIndex` - `eth_feeHistory` - `eth_getLogs` .Plus following methods are answered directly by Dshackle - `net_version` - `net_peerCount` - `net_listening` - `web3_clientVersion` - `eth_protocolVersion` - `eth_syncing` - `eth_coinbase` - `eth_mining` - `eth_hashrate` - `eth_accounts` It's possible to enable additional methods that are available on upstream, or disable an existing method. For that purpose there is `methods` configuration: [source, yaml] ---- upstreams: - id: my-node chain: ethereum labels: archive: true methods: enabled: - name: trace_transaction disabled: - name: eth_getBlockByNumber ---- Such configuration option allows executing method `trace_transaction` and also disables `eth_getBlockByNumber` on that particular upstream. If a client requests to execute method `trace_transaction` then it will be scheduled to that upstream (or any upstream with such method enabled). NOTE: It's especially useful when used together with upstream labels.If an archive upstream has label `archive: true` it's possible to specify that the client wants to execute method `trace_transaction` only on an archive node(s), which has complete historical data for tracing. === Static Methods You can overwrite existing methods or add new ones using a static response: [source,yaml] ---- upstreams: - id: my-node chain: ethereum methods: enabled: - name: net_version static: "\"100000\"" - name: eth_chainId static: "0x186a0" - name: eth_custom_array static: '["custom_array_response"]' - name: eth_custom_bool static: "false" ---- === Authentication ==== TLS All connection types can use TLS secured connection, with optional client certificate authentication: - `ca` path to certificate required from remote server - optional `certificate` and `key` for client authentication. NOTE: Please note that `key` must be encoded with _PKCS 8_ ==== Basic Authentication For JSON RPC and Websockets a Basic Authentication can be used: - `username` - username - `password` - password