aristotle @ rpc-de-32 crash-looped after its build was fixed:
priv_validator_state.json found in /root/.0g. Continuing!
Already initialized, continuing!
panic: ProvideBlsSigner (node-core/components/signer.go:46):
key file does not exist at path: /root/.0g/config/priv_validator_key.json
$DATA_DIR and $CONFIG_DIR are SEPARATE docker volumes (<node>_zerog and
<node>_config), but the guard tested only $DATA_DIR/priv_validator_state.json —
inferring the state of the config volume from a file on the data volume. Worse, the
copy order wrote that guard file BEFORE the two key files, so an init interrupted
between them left the data volume with the guard and the config volume without the
keys. Every later start then took the 'Already initialized' branch, copied nothing,
and panicked. Unrecoverable by restarting: the node wedges permanently. aristotle
got there via the repeated build failures earlier today.
Fix, two parts:
- guard on everything 0gchaind needs to boot (both config keys AND the data state
file), not one proxy file on the wrong volume;
- write the keys FIRST and the guard file LAST, so an interrupted init re-runs on
the next start instead of latching into the wedged state.
Tested locally against the real 0gchaind binary (aristotle v1.0.6), sandboxed HOME:
scenario before after
fresh volumes keys created keys created (no regression)
wedged (state, no keys) keys MISSING keys created (unwedges)
partial (state+node_key) keys MISSING keys created
healthy (all present) no re-init no re-init, and a SENTINEL written
into priv_validator_key.json survives
-> never clobbers an initialized node
The 'wedged' row reproduces aristotle's production failure exactly.
Note this regenerates node identity on an affected node (node_key.json,
priv_validator_key.json). These are RPC nodes that never sign, so a fresh identity is
harmless; chain data on the _zerog volume is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the de-32 build break introduced by vibe-node #2912 (galileo 3.0.3 -> 3.0.8,
merged as bc4cecf). Job ad580587:
#16 chmod: cannot access '/0g/bin/0gchaind': No such file or directory
The Dockerfile hardcoded 'galileo -> mv <root>/rpc /0g'. That held for 3.0.3, where
every profile dir carried its own copy of the binaries. 3.0.8 DEDUPLICATED them into
a shared <root>/bin/ (168 MB -> 69 MB), leaving rpc/ as configs-only — so the move
produced configs and no binaries, and the build died two layers later on a chmod.
Note 'does <root>/rpc exist' is not a sufficient test: 3.0.8 still ships rpc/, it just
no longer holds bin/. Verified layouts:
galileo 3.0.3 <root>/{rpc,validator,archive,seed}/bin/ + configs
galileo 3.0.8 <root>/bin/ shared + <root>/{rpc,...}/ configs
aristotle 1.0.6 <root>/bin/ + configs, no profile dirs
So: take configs from the profile dir when present, then top the binaries up from the
shared root bin/ if the profile dir did not supply them.
Also asserts both binaries exist immediately after extraction, naming the chain, the
version and the known layouts — so the next upstream reshuffle reports itself at the
point of failure rather than as an anonymous chmod error.
Verified against the real artifacts — all three now yield an identical /0g
(bin/0gchaind 76M, bin/geth 46M, geth-config.toml, geth-genesis.json,
kzg-trusted-setup.json, 0g-home/, rollback_cl.sh), and a synthetic unknown layout
exits 1 with the diagnostic instead of reaching chmod.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0G publishes aristotle and galileo from two different repos, and neither uses a
consistent release-tag format. The asset filename is always <spec>-v<version>.tar.gz;
only the tag varies:
aristotle -> 0gchain-Aristotle 1.0.4 = "1.0.4" 1.0.6 = "v1.0.6"
galileo -> 0gchain-NG 3.0.3 = "v3.0.3" 3.0.7 = "galileo-v3.0.7" 3.0.8 = "v3.0.8"
The hardcoded formats meant aristotle could not go past 1.0.4 and galileo could not
reach 3.0.7. Resolve the tag by trying the known forms in order instead.
Also switch curl -sL to -fsSL. Without -f a 404 HTML body was written into the
.tar.gz and the build failed several layers later at 'tar: not in gzip format',
an error pointing nowhere near the cause. That has now caused three separate
misdiagnoses, including a de-32 build blocker.
Verified end-to-end against the real artifacts: aristotle 1.0.4/1.0.6 and galileo
3.0.3/3.0.7/3.0.8 all resolve; aristotle 3.0.3 and galileo 9.9.9 fail immediately
with an actionable message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add zircuit-mainnet to conduit-op-reth (ghcr.io/conduitxyz/conduit-op-reth:v2.1.1)
- Add genesis.json/rollup.json from Conduit API with bedrockBlock=32956468 fix
- Add Conduit P2P bootnodes + static peer
- Use ETHEREUM_MAINNET_* vars for L1 (chainid 1) with oplabs op-node:v1.19.3
- Remove zircuit-mainnet op-geth archive/pruned profiles (SLS dependency dead)
- Keep sequencer endpoint and drpc_chain=zircuit-mainnet
SLS-decommission root cause: zircuit1/l2-geth hard-requires Zircuit-hosted
SLS DB that no longer resolves (mainnet-sls-db-global-primary.cluster-ro...
no such host). Migrating to Conduit external-node stack per operator directive.
Restore the beacon x-upstream options.disable-validation: true that was
removed during regeneration. This prevents dshackle 0.79.10's beacon health
validator from timing out on the spec-compliant empty-body 200 that nimbus
returns, causing the upstream to flap UNAVAILABLE. Matches the established
ethereum-beacon pattern.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Maru CL stuck behind tip was sending FCU with finalized ahead of EL head
(Besu -38002). Comment out [linea] until EL catches tip (no enabled flag).
Co-authored-by: Cursor <cursoragent@cursor.com>
ss1 shipped without drpcorg public-submodule resources and crash-loops
at boot (compatible-clients.yaml missing); ss2 is the boot-verified fix.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fleet-wide default is now stakesquid/dshackle:0.79.11-ss1 (digest-pinned);
DSHACKLE_IMAGE env escape hatch retained for rollback to upstream.
Co-authored-by: Cursor <cursoragent@cursor.com>
Default remains drpcorg/dshackle:0.79.10; canary hosts can set
DSHACKLE_IMAGE via host_vars env_overrides without editing compose.
Co-authored-by: Cursor <cursoragent@cursor.com>
Orphaned op-geth files (already removed from config.yml) plus sibling
op-reth/op-erigon profiles still share Traefik service names/paths under
COMPOSE_FILE. Keep only ink-mainnet-op-reth-pruned-trace--kona.yml as the
single active /ink-mainnet route.
EL/CL baseFee mismatch at block 22961752; CL crash-loops on
INVALID payload from post-v1.4.2 block format while beacon-kit
is on v1.4.1. Also regenerates ink-mainnet OPE P2P bootnodes
from parent a790c3e and refreshes reference-rpc-endpoint (mova).
Co-authored-by: Cursor <cursoragent@cursor.com>
Image default is -Xms1024M with no -Xmx, so heap can grow unbounded (~30GiB).
us-41 was at 90% memory with no swap; -Xmx4g provides a safety limit.
Co-authored-by: Cursor <cursoragent@cursor.com>
Canonicalize plasma-testnet so the tracked compose adopts the same
volume/service names as the de-32 hand-written override (rpc_plasma-
testnet-reth-pruned-trace-{execution,consensus,jwt}). Also refresh
mainnet observer sed to be chain-agnostic.
Also regenerate compose for already-merged hashkey static-peers / op-geth
static-nodes entrypoint (parent #1437) that had not yet landed in rpc.
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace buggy && short-circuit with if/fi so the one-shot init
exits 0 when the signing key is already present (matches mainnet/
moderato and templates/clients/reth.tempo.yml), unblocking sync
resume on data-intact hosts still on the orphaned testnet compose.
Insert validate-dshackle-1to1.py between the overrides weave and the HUP signal:
a dshackle config routing >1 node for the same chain is REFUSED (exit 1) so the
live config stays active. Structural enforcement of 1 dshackle = 1 node/chain —
we can't attribute traffic to multiple nodes behind one proxy (no per-upstream
request metric; conn-seconds biases it). See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Refuses to activate a dshackle config routing >1 upstream for the same chain
(any kind/client) — we cannot attribute traffic to multiple nodes behind one
proxy (no per-upstream request-count metric; conn-seconds biases it), and the
attribution model + placement planner both assume 1:1. Wire into
reload_dshackle.sh BEFORE the reload so a drifting config can never go live.
Host-local by design (the repo doesn't track host->dshackle assignment; 1:1 is
only checkable against the assembled dshackle config at /root/rpc/main_configs).
See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Empty ws-origins rejects WebSocket connections with an Origin header
(dshackle head-subscription monitoring). Sed under [json-rpc] like
address/ws-address so restarts keep the value.
claude
