aristotle @ rpc-de-32 crash-looped after its build was fixed:
priv_validator_state.json found in /root/.0g. Continuing!
Already initialized, continuing!
panic: ProvideBlsSigner (node-core/components/signer.go:46):
key file does not exist at path: /root/.0g/config/priv_validator_key.json
$DATA_DIR and $CONFIG_DIR are SEPARATE docker volumes (<node>_zerog and
<node>_config), but the guard tested only $DATA_DIR/priv_validator_state.json —
inferring the state of the config volume from a file on the data volume. Worse, the
copy order wrote that guard file BEFORE the two key files, so an init interrupted
between them left the data volume with the guard and the config volume without the
keys. Every later start then took the 'Already initialized' branch, copied nothing,
and panicked. Unrecoverable by restarting: the node wedges permanently. aristotle
got there via the repeated build failures earlier today.
Fix, two parts:
- guard on everything 0gchaind needs to boot (both config keys AND the data state
file), not one proxy file on the wrong volume;
- write the keys FIRST and the guard file LAST, so an interrupted init re-runs on
the next start instead of latching into the wedged state.
Tested locally against the real 0gchaind binary (aristotle v1.0.6), sandboxed HOME:
scenario before after
fresh volumes keys created keys created (no regression)
wedged (state, no keys) keys MISSING keys created (unwedges)
partial (state+node_key) keys MISSING keys created
healthy (all present) no re-init no re-init, and a SENTINEL written
into priv_validator_key.json survives
-> never clobbers an initialized node
The 'wedged' row reproduces aristotle's production failure exactly.
Note this regenerates node identity on an affected node (node_key.json,
priv_validator_key.json). These are RPC nodes that never sign, so a fresh identity is
harmless; chain data on the _zerog volume is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the de-32 build break introduced by vibe-node #2912 (galileo 3.0.3 -> 3.0.8,
merged as bc4cecf). Job ad580587:
#16 chmod: cannot access '/0g/bin/0gchaind': No such file or directory
The Dockerfile hardcoded 'galileo -> mv <root>/rpc /0g'. That held for 3.0.3, where
every profile dir carried its own copy of the binaries. 3.0.8 DEDUPLICATED them into
a shared <root>/bin/ (168 MB -> 69 MB), leaving rpc/ as configs-only — so the move
produced configs and no binaries, and the build died two layers later on a chmod.
Note 'does <root>/rpc exist' is not a sufficient test: 3.0.8 still ships rpc/, it just
no longer holds bin/. Verified layouts:
galileo 3.0.3 <root>/{rpc,validator,archive,seed}/bin/ + configs
galileo 3.0.8 <root>/bin/ shared + <root>/{rpc,...}/ configs
aristotle 1.0.6 <root>/bin/ + configs, no profile dirs
So: take configs from the profile dir when present, then top the binaries up from the
shared root bin/ if the profile dir did not supply them.
Also asserts both binaries exist immediately after extraction, naming the chain, the
version and the known layouts — so the next upstream reshuffle reports itself at the
point of failure rather than as an anonymous chmod error.
Verified against the real artifacts — all three now yield an identical /0g
(bin/0gchaind 76M, bin/geth 46M, geth-config.toml, geth-genesis.json,
kzg-trusted-setup.json, 0g-home/, rollback_cl.sh), and a synthetic unknown layout
exits 1 with the diagnostic instead of reaching chmod.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0G publishes aristotle and galileo from two different repos, and neither uses a
consistent release-tag format. The asset filename is always <spec>-v<version>.tar.gz;
only the tag varies:
aristotle -> 0gchain-Aristotle 1.0.4 = "1.0.4" 1.0.6 = "v1.0.6"
galileo -> 0gchain-NG 3.0.3 = "v3.0.3" 3.0.7 = "galileo-v3.0.7" 3.0.8 = "v3.0.8"
The hardcoded formats meant aristotle could not go past 1.0.4 and galileo could not
reach 3.0.7. Resolve the tag by trying the known forms in order instead.
Also switch curl -sL to -fsSL. Without -f a 404 HTML body was written into the
.tar.gz and the build failed several layers later at 'tar: not in gzip format',
an error pointing nowhere near the cause. That has now caused three separate
misdiagnoses, including a de-32 build blocker.
Verified end-to-end against the real artifacts: aristotle 1.0.4/1.0.6 and galileo
3.0.3/3.0.7/3.0.8 all resolve; aristotle 3.0.3 and galileo 9.9.9 fail immediately
with an actionable message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Add zircuit-mainnet to conduit-op-reth (ghcr.io/conduitxyz/conduit-op-reth:v2.1.1)
- Add genesis.json/rollup.json from Conduit API with bedrockBlock=32956468 fix
- Add Conduit P2P bootnodes + static peer
- Use ETHEREUM_MAINNET_* vars for L1 (chainid 1) with oplabs op-node:v1.19.3
- Remove zircuit-mainnet op-geth archive/pruned profiles (SLS dependency dead)
- Keep sequencer endpoint and drpc_chain=zircuit-mainnet
SLS-decommission root cause: zircuit1/l2-geth hard-requires Zircuit-hosted
SLS DB that no longer resolves (mainnet-sls-db-global-primary.cluster-ro...
no such host). Migrating to Conduit external-node stack per operator directive.
Restore the beacon x-upstream options.disable-validation: true that was
removed during regeneration. This prevents dshackle 0.79.10's beacon health
validator from timing out on the spec-compliant empty-body 200 that nimbus
returns, causing the upstream to flap UNAVAILABLE. Matches the established
ethereum-beacon pattern.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
bor snapshot prune-block wants the directory holding the freezer tables
(bodies.*.cdat) - the modern layout splits ancient/ into chain/ + state/,
so ancient/ root fails with 'ancient chain segments already extracted' and
the prune container crash-loops (de-12, 2026-08-02). Matches BSC's
prune-block usage which targets .../ancient/chain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Maru CL stuck behind tip was sending FCU with finalized ahead of EL head
(Besu -38002). Comment out [linea] until EL catches tip (no enabled flag).
Co-authored-by: Cursor <cursoragent@cursor.com>
A 0G sidecar volume (ronin eigenda-proxy) FATAL'd the whole backup before the
manifest was written - verify found no new backup and the refresh pipeline's
bench-retire escalated 4 cycles in a row, head-of-line blocking the single
refresh lane. Sidecars are legitimately tiny; the <1G fail-closed floor now
applies only to the main data volume (key == compose basename).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ss1 shipped without drpcorg public-submodule resources and crash-loops
at boot (compatible-clients.yaml missing); ss2 is the boot-verified fix.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fleet-wide default is now stakesquid/dshackle:0.79.11-ss1 (digest-pinned);
DSHACKLE_IMAGE env escape hatch retained for rollback to upstream.
Co-authored-by: Cursor <cursoragent@cursor.com>
Default remains drpcorg/dshackle:0.79.10; canary hosts can set
DSHACKLE_IMAGE via host_vars env_overrides without editing compose.
Co-authored-by: Cursor <cursoragent@cursor.com>
HSK's published mainnet genesis sets isthmusTime but omits pragueTime; op-geth's
OP-Stack validation rejects it ('PragueTime (<nil>) must equal IsthmusTime'), so
geth init aborts and the EL runs on a stale chainconfig. testnet genesis correctly
sets pragueTime=isthmusTime. Adding it here makes init pass. (Runtime fork activation
is handled by GETH_OVERRIDE_* in vibe-node geth.env; this fixes init + matches testnet.)
Co-Authored-By: Claude <noreply@anthropic.com>
Orphaned op-geth files (already removed from config.yml) plus sibling
op-reth/op-erigon profiles still share Traefik service names/paths under
COMPOSE_FILE. Keep only ink-mainnet-op-reth-pruned-trace--kona.yml as the
single active /ink-mainnet route.
