Operator ruling: NEVER force a reconnect on a routine reload/rpc-update run.
Only upstream REMOVALS or new disabled-method ADDITIONS require a gateway
reconnect to be seen by dRPC gateways.
Implement snapshot-based comparison:
- Before SIGHUP, read current advertised set (upstream IDs + disabled methods)
- Compare against previous snapshot (if exists)
- Drop established gRPC connections ONLY IF:
* an upstream ID disappeared, OR
* a new disabled-method entry appeared
- Pure additions or no change: reload only, no drop
- First run with no prior snapshot: do NOT drop, just write snapshot
- Always rewrite snapshot after successful reload
- All operations best-effort (failures never fail the script)
Live-measured: upstream/method additions propagate over existing connection
in ~16s (no reconnect needed).
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Fix AVAGO_STAKING_PORT to use fixed port 9651 instead of salted client_port.
Fix port mapping from {{ client_port }}:{{ client_port }} to {{ client_port }}:9651
to ensure container uses fixed P2P port while host binds to salted port.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Extend reload_dshackle.sh to drop established gateway gRPC connections on
port 2449 after a successful SIGHUP config reload, forcing the dRPC edge
to reconnect within ~1s and re-read the advertised chain/method list.
Gateways only re-initialize advertisements on reconnection (verified live:
0.03-0.06s serving gap, 33 re-probes in 4s). This allows deploy-time upstream
additions (new chains like mova, new backends like us-50) to be seen by
dRPC immediately rather than waiting for natural reconnection.
The TCP-level drop is best-effort: failure does NOT fail the script.
Operator ruling: this stays a TCP-level drop, NOT a dshackle code change.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add FEEHISTORY_CAP env tunable with default 250
- Update init.sh to use FEEHISTORY_CAP for feehistory-cap in app.toml
- Maintain backward compatibility with JSON_RPC_FEEHISTORY_CAP
- Apply to both mainnet and testnet profiles
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Matches the run profile flags to prevent 'incompatible state scheme' crash
on fresh datadir initialization (stored: hash, provided: path).
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Fix ABCORE_MAINNET_GETH_VERSION default from empty to v1.13.15-abcore-1.2
- Fix command flag from '--config /config/config.toml' to '--config=/config/config.toml'
Both issues caused first-deploy failures on rpc-de-32:
1. Empty version default caused curl exit 22 on binary download
2. Space-separated flag/value caused geth crash-loop with 'flag provided but not defined: -config'
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add abcore.Dockerfile for geth-fork binary download from ABFoundationGlobal/abcore
- Add scripts/init.sh for geth initialization with genesis.json and config.toml
- Add generated compose file abcore-mainnet-geth-pruned-pebble-path.yml
- Update compose_registry.json with new entry
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
The previous commit (7cbd0532) added VERSION normalization to the DOWNLOAD
RUN step but forgot to add it to the EXTRACTION RUN step. Each Dockerfile
RUN is a fresh shell, so the extraction step fails with 'VERSION: unbound
variable' when it tries to use ${VERSION} in the SRC variable.
Fix: add VERSION="${ZERO_GRAVITY_VERSION#v}"; to the extraction RUN step
(line 73) so it defines VERSION before using it.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Fix four sed address ranges that were missing closing slash before comma:
- /^\[rpc\],/^\[/ -> /^\[rpc\]/,/^\[/
- /^\[json-rpc\],/^\[/ -> /^\[json-rpc\]/,/^\[/ (3 occurrences)
The malformed patterns caused sed to crash-loop the TAC container.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Match upstream release tags: aristotle v1.0.6 (0gchain-Aristotle) and
galileo v3.0.8 (0gchain-NG) both carry v-prefix in their release tags.
Required for 0g hardfork; old clients stuck-syncing.
Supersedes stale unmergeable PR ethereum-rpc-docker#62.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
The TAC mainnet container was crash-looping with 'sed: unsupported command ^'
because busybox sed in Alpine does not support the GNU sed extensions used in
the cometbft-common.sh scripts (address ranges with ^ anchor and {} grouping).
Adding the 'sed' package installs GNU sed which supports these features.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add archive profile (pruning=nothing) for tacchaind
- Change client_rpc_port from 26657 (CometBFT) to 8545 (EVM JSON-RPC)
- Update Dockerfile to download pre-built binary from GitHub releases v1.6.0
- Verify binary against checksums.txt
- Update init.sh to support PRUNING env var (nothing for archive, default for pruned)
- Update peers from NETWORKS.md
- Add archive snapshot URL note
- Traefik routes to EVM JSON-RPC port 8545 for both pruned and archive
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
show-status.sh could never report failure. The ansible task that wraps it
('Execute show-status.sh and fail on failure') therefore always passed, on every
host, regardless of node state. Three bugs, each masking the next:
1. $? read too late. `code=0` sits between the sync-status.sh call and
`if [ $? -ne 0 ]`. A plain assignment succeeds and overwrites $? with 0, so
the condition was ALWAYS false and the else branch always taken.
2. The else branch was inverted. It is the sync-status-SUCCEEDED path, yet it set
`code=1; any_failure=true` — marking healthy nodes as failures.
3. any_failure could never propagate. check_sync_status runs backgrounded (`&`),
i.e. in a subshell, so `any_failure=true` inside it is discarded; and the
`wait "$pid"` loop threw away each job's exit status.
(3) hid (1) and (2): a script that believed every node had failed still exited 0,
so nobody saw it.
Fix: capture rc immediately; restore the intended logic (success => 0, syncing or
lagging => tolerated, anything else => failure); propagate failure in the PARENT
via `wait "$pid" || any_failure=true`, since the subshell cannot.
Verified with a stubbed sync-status.sh:
scenario before after
all online 0 0
one syncing 0 0 (tolerated)
one lagging 0 0 (tolerated)
one ERROR 0 1
ALL error 0 1
Behaviour for healthy fleets is unchanged; only genuine failures now surface.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
aristotle @ rpc-de-32 crash-looped after its build was fixed:
priv_validator_state.json found in /root/.0g. Continuing!
Already initialized, continuing!
panic: ProvideBlsSigner (node-core/components/signer.go:46):
key file does not exist at path: /root/.0g/config/priv_validator_key.json
$DATA_DIR and $CONFIG_DIR are SEPARATE docker volumes (<node>_zerog and
<node>_config), but the guard tested only $DATA_DIR/priv_validator_state.json —
inferring the state of the config volume from a file on the data volume. Worse, the
copy order wrote that guard file BEFORE the two key files, so an init interrupted
between them left the data volume with the guard and the config volume without the
keys. Every later start then took the 'Already initialized' branch, copied nothing,
and panicked. Unrecoverable by restarting: the node wedges permanently. aristotle
got there via the repeated build failures earlier today.
Fix, two parts:
- guard on everything 0gchaind needs to boot (both config keys AND the data state
file), not one proxy file on the wrong volume;
- write the keys FIRST and the guard file LAST, so an interrupted init re-runs on
the next start instead of latching into the wedged state.
Tested locally against the real 0gchaind binary (aristotle v1.0.6), sandboxed HOME:
scenario before after
fresh volumes keys created keys created (no regression)
wedged (state, no keys) keys MISSING keys created (unwedges)
partial (state+node_key) keys MISSING keys created
healthy (all present) no re-init no re-init, and a SENTINEL written
into priv_validator_key.json survives
-> never clobbers an initialized node
The 'wedged' row reproduces aristotle's production failure exactly.
Note this regenerates node identity on an affected node (node_key.json,
priv_validator_key.json). These are RPC nodes that never sign, so a fresh identity is
harmless; chain data on the _zerog volume is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the de-32 build break introduced by vibe-node #2912 (galileo 3.0.3 -> 3.0.8,
merged as bc4cecf). Job ad580587:
#16 chmod: cannot access '/0g/bin/0gchaind': No such file or directory
The Dockerfile hardcoded 'galileo -> mv <root>/rpc /0g'. That held for 3.0.3, where
every profile dir carried its own copy of the binaries. 3.0.8 DEDUPLICATED them into
a shared <root>/bin/ (168 MB -> 69 MB), leaving rpc/ as configs-only — so the move
produced configs and no binaries, and the build died two layers later on a chmod.
Note 'does <root>/rpc exist' is not a sufficient test: 3.0.8 still ships rpc/, it just
no longer holds bin/. Verified layouts:
galileo 3.0.3 <root>/{rpc,validator,archive,seed}/bin/ + configs
galileo 3.0.8 <root>/bin/ shared + <root>/{rpc,...}/ configs
aristotle 1.0.6 <root>/bin/ + configs, no profile dirs
So: take configs from the profile dir when present, then top the binaries up from the
shared root bin/ if the profile dir did not supply them.
Also asserts both binaries exist immediately after extraction, naming the chain, the
version and the known layouts — so the next upstream reshuffle reports itself at the
point of failure rather than as an anonymous chmod error.
Verified against the real artifacts — all three now yield an identical /0g
(bin/0gchaind 76M, bin/geth 46M, geth-config.toml, geth-genesis.json,
kzg-trusted-setup.json, 0g-home/, rollback_cl.sh), and a synthetic unknown layout
exits 1 with the diagnostic instead of reaching chmod.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0G publishes aristotle and galileo from two different repos, and neither uses a
consistent release-tag format. The asset filename is always <spec>-v<version>.tar.gz;
only the tag varies:
aristotle -> 0gchain-Aristotle 1.0.4 = "1.0.4" 1.0.6 = "v1.0.6"
galileo -> 0gchain-NG 3.0.3 = "v3.0.3" 3.0.7 = "galileo-v3.0.7" 3.0.8 = "v3.0.8"
The hardcoded formats meant aristotle could not go past 1.0.4 and galileo could not
reach 3.0.7. Resolve the tag by trying the known forms in order instead.
Also switch curl -sL to -fsSL. Without -f a 404 HTML body was written into the
.tar.gz and the build failed several layers later at 'tar: not in gzip format',
an error pointing nowhere near the cause. That has now caused three separate
misdiagnoses, including a de-32 build blocker.
Verified end-to-end against the real artifacts: aristotle 1.0.4/1.0.6 and galileo
3.0.3/3.0.7/3.0.8 all resolve; aristotle 3.0.3 and galileo 9.9.9 fail immediately
with an actionable message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add zircuit-mainnet to conduit-op-reth (ghcr.io/conduitxyz/conduit-op-reth:v2.1.1)
- Add genesis.json/rollup.json from Conduit API with bedrockBlock=32956468 fix
- Add Conduit P2P bootnodes + static peer
- Use ETHEREUM_MAINNET_* vars for L1 (chainid 1) with oplabs op-node:v1.19.3
- Remove zircuit-mainnet op-geth archive/pruned profiles (SLS dependency dead)
- Keep sequencer endpoint and drpc_chain=zircuit-mainnet
SLS-decommission root cause: zircuit1/l2-geth hard-requires Zircuit-hosted
SLS DB that no longer resolves (mainnet-sls-db-global-primary.cluster-ro...
no such host). Migrating to Conduit external-node stack per operator directive.
Restore the beacon x-upstream options.disable-validation: true that was
removed during regeneration. This prevents dshackle 0.79.10's beacon health
validator from timing out on the spec-compliant empty-body 200 that nimbus
returns, causing the upstream to flap UNAVAILABLE. Matches the established
ethereum-beacon pattern.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Maru CL stuck behind tip was sending FCU with finalized ahead of EL head
(Besu -38002). Comment out [linea] until EL catches tip (no enabled flag).
Co-authored-by: Cursor <cursoragent@cursor.com>
ss1 shipped without drpcorg public-submodule resources and crash-loops
at boot (compatible-clients.yaml missing); ss2 is the boot-verified fix.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fleet-wide default is now stakesquid/dshackle:0.79.11-ss1 (digest-pinned);
DSHACKLE_IMAGE env escape hatch retained for rollback to upstream.
Co-authored-by: Cursor <cursoragent@cursor.com>
Default remains drpcorg/dshackle:0.79.10; canary hosts can set
DSHACKLE_IMAGE via host_vars env_overrides without editing compose.
Co-authored-by: Cursor <cursoragent@cursor.com>
Orphaned op-geth files (already removed from config.yml) plus sibling
op-reth/op-erigon profiles still share Traefik service names/paths under
COMPOSE_FILE. Keep only ink-mainnet-op-reth-pruned-trace--kona.yml as the
single active /ink-mainnet route.
EL/CL baseFee mismatch at block 22961752; CL crash-loops on
INVALID payload from post-v1.4.2 block format while beacon-kit
is on v1.4.1. Also regenerates ink-mainnet OPE P2P bootnodes
from parent a790c3e and refreshes reference-rpc-endpoint (mova).
Co-authored-by: Cursor <cursoragent@cursor.com>
Image default is -Xms1024M with no -Xmx, so heap can grow unbounded (~30GiB).
us-41 was at 90% memory with no swap; -Xmx4g provides a safety limit.
Co-authored-by: Cursor <cursoragent@cursor.com>
Canonicalize plasma-testnet so the tracked compose adopts the same
volume/service names as the de-32 hand-written override (rpc_plasma-
testnet-reth-pruned-trace-{execution,consensus,jwt}). Also refresh
mainnet observer sed to be chain-agnostic.
claudeandMistral Vibe