= Emerald Dshackle
:imagesdir: docs/assets
ifdef::env-github[]
:imagesdir: https://raw.githubusercontent.com/emeraldpay/dshackle/master/docs/assets
endif::[]
image:https://img.shields.io/docker/pulls/emeraldpay/dshackle?style=flat-square["Docker", link="https://hub.docker.com/r/emeraldpay/dshackle"]
image:https://img.shields.io/github/license/emeraldpay/dshackle.svg?style=flat-square&maxAge=2592000["License", link="https://github.com/emeraldpay/dshackle/blob/master/LICENSE"]
image:https://badges.gitter.im/emeraldpay/community.svg[link="https://gitter.im/emeraldpay/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge]
Dshackle is an L7 Blockchain API Load Balancer with automatic discovery and health checking, authentication, and TLS termination.
It is designed to work as an edge proxy, middle proxy, or API gateway.
Dshackle provided a high level aggregated API on top of several underlying upstreams (blockchain nodes or providers, such
as Geth, Parity, Infura, etc), it automatically verifies their availability and the current status of the network, executes
commands making sure that the response is consistent and/or data successfully broadcasted to the network.
The main goals of the project is to:
- separate application services and blockchain nodes
- make blockchain access compatible with modern microservice oriented architecture
- provide secure access to a blockchain, on both protocol and data level
The main features and advantages are:
- leveraging gRPC and HTTP2 protocols, with server push and asynchronous communications, to simplify and optimize standard
access patterns
- targeting Kubernetes architecture
- automatically distributing access to API through multiple different target nodes, taking into account their current
availability and status
- allowing to build a mesh network of routers in different regions sharing a set of underlying nodes, with automatic
rebalancing and smart routing
- caching data on the edge
- providing monitoring (ex. Prometheus) and externalizable logging
- configurable access authentication and authorization, including TLS certificates
Dshackle connects to several upstreams via JSON RPC, Websockets, or gRPC protocols. It verifies if a node ("upstream") is
fully synchronized (not in initial sync mode), has enough peers, and its height is not behind other nodes. If upstream lags
behind others, lost peers, started to resync or went down, then Dshackle temporarily excludes it from requests and returns
when the upstream problem is fixed.
image::call-schema.png[alt="Call Schema",width=100%,align="center"]
== Quick Start
=== Configuration
Create file `dshackle.yaml` with following content:
[source,yaml]
----
version: v1
port: 2449
tls:
enabled: false
upstreams:
config: "upstreams.yaml"
----
Which sets the following:
- application listen on 0.0.0.0:2449
- TLS security is disabled (_don't use in production!_)
- read upstreams configuration from file `upstreams.yaml` in the current directory
Now create file `upstreams.yaml`:
[source,yaml]
----
version: v1
upstreams:
- id: infura-eth
chain: ethereum
connection:
ethereum:
rpc:
url: "https://mainnet.infura.io/v3/${INFURA_USER}"
ws:
url: "wss://mainnet.infura.io/ws/v3/${INFURA_USER}"
- id: infura-kovan
chain: kovan
connection:
ethereum:
rpc:
url: "https://kovan.infura.io/v3/${INFURA_USER}"
----
This configures:
- setups 2 upstreams, one for Ethereum Mainnet and another for Kovan Testnet (both upstreams are configured to use Infura endpoint)
- for Ethereum Mainnet it connects using JSON RPC and Websockets connections, for Kovan just JSON RPC is used
- Infura authentication config is omitted for this demo
- `${INFURA_USER}` will be provided through environment variable
==== Run docker image
Official Docker image you can find at: emeraldpay/dshackle
.Setup Infura username
[source,bash]
----
export INFURA_USER=...
----
.Run Dshackle
[source,bash]
----
docker run -p 2449:2449 -v $(pwd):/etc/dshackle -e "INFURA_USER=$INFURA_USER" emeraldpay/dshackle
----
Now it listen on port 2449 at the localhost and can be connected from any gRPC compatible client.
Tools such as https://github.com/fullstorydev/grpcurl[gRPCurl] can automatically parse protobuf definitions and connect
to it (actual Protobuf sources are located in a separate repository which you can find at https://github.com/emeraldpay/proto)
.Connect and listen for new blocks on Ethereum Mainnet
[source,bash]
----
grpcurl -import-path ./proto/ -proto blockchain.proto -d "{\"type\": 100}" -plaintext 127.0.0.1:2449 io.emeraldpay.api.Blockchain/SubscribeHead
----
.Output would be like
----
{
"chain": "CHAIN_ETHEREUM",
"height": 8396159,
"blockId": "fc58a258adccc94466ae967b1178eea721349b0667f59d5fe1b0b436460bce75",
"timestamp": 1566423564000,
"weight": "AnMcf2VJB5kOSQ=="
}
{
"chain": "CHAIN_ETHEREUM",
"height": 8396160,
"blockId": "787899711b862b77df8d2faa69de664048598265a9f96abf178d341076e200e0",
"timestamp": 1566423574000,
"weight": "AnMch35tO6hSGg=="
}
...
...
----
The output above is for a _streaming subscription_ to all new blocks on Ethereum Mainnet. It's one of services provided
by Dshackle, in additional to standard methods provided by RPC JSON of underlying nodes.
== Documentation
For detailed documentation see link:docs/[] directory.
== Client Libraries
=== Java gRPC Client
image:https://api.bintray.com/packages/emerald/emerald-grpc/emerald-grpc/images/download.svg[link="https://bintray.com/emerald/emerald-grpc/emerald-grpc/"]
https://github.com/emeraldpay/emerald-java-client
[source,groovy]
----
repositories {
maven {
url "https://dl.bintray.com/emerald/emerald-grpc"
}
}
dependencies {
compile "io.emeraldpay:emerald-grpc:0.6.0-0.2"
}
----
=== Javascript gRPC Client
image:https://img.shields.io/npm/v/@emeraldpay/grpc-client.svg["npm (scoped)", link="https://www.npmjs.com/package/@emeraldpay/grpc-client"]
https://github.com/emeraldpay/emerald-js-grpc
[source,json]
----
"dependencies": {
"@emeraldpay/grpc-client": "0.11.0-0.2",
}
----
See more in the documentation for link:docs/10-client-libraries.adoc[Client Libraries].
== Development
=== Setting up environment
Dshackle is JVM based project written in Kotlin. To build and run it from sources you'll need to install
https://openjdk.java.net/projects/jdk/11/[Java JDK] and https://gradle.org/[Gradle]
NOTE: For the current `master` branch you'll need to install a development version of Etherjar
----
git clone git@github.com:infinitape/etherjar.git
cd etherjar
gradle install
----
=== Build Dshackle
==== Build everything
[source, bash]
----
gradle build
----
==== Make a Zip distribution
[source, bash]
----
gradle distZip
----
You can find a redistributable zip in `build/distributions`
==== Make a Docker distribution
[source, bash]
----
gradle jib -Pdocker=gcr.io/myproject
----
Gradle will prepare a Docker image and upload it to your custom Docker Registry at `gcr.io/myproject` (please change to address of your actual registry)
==== Architecture
Dshackle is built using:
- Kotlin
- Spring Framework + Spring Boot
- Spring Reactor
- Netty
- Etherjar
- gRPC and HTTP2 protocol
- Groovy and Spock for testing
== Community
=== Development Chat
image:https://badges.gitter.im/emeraldpay/community.svg[link="https://gitter.im/emeraldpay/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge]
== Commercial Support
Want to support the project, prioritize a specific feature, or get commercial help with using Dshackle in your project?
Please contact splix@emeraldpay.io to discuss the possibility
== License
Copyright 2019 ETCDEV GmbH
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.