zero-gravity: init guard must check the CONFIG volume, not a data-volume proxy #64

claude · 2026-08-12T19:34:15Z

claude commented

2026-08-12 19:34:15 +00:00

ASSET-only — zero-gravity/scripts/init.sh. Script-only, so no ./update.sh and no gitlink change.

Symptom

aristotle @ rpc-de-32 crash-loops. show-status reports it as syncing, which is wrong — the CL never starts:

priv_validator_state.json found in /root/.0g. Continuing!
Already initialized, continuing!
panic: error calling provider ProvideBlsSigner (node-core/components/signer.go:46):
       key file does not exist at path: /root/.0g/config/priv_validator_key.json

The EL is a casualty, not the cause — it sits at peercount=0 with zero engine traffic because nothing is driving it.

Root cause: cross-volume inference, plus a write order that latches the failure

$DATA_DIR and $CONFIG_DIR are separate docker volumes (<node>_zerog → /root/.0g/data, <node>_config → /root/.0g/config). The guard tested only:

if [ ! -f "$DATA_DIR/priv_validator_state.json" ]; then

— inferring the state of the config volume from a file on the data volume. And the copies wrote that guard file before the keys:

cp .../priv_validator_state.json $DATA_DIR   # 2  <- guard file
cp .../node_key.json             $CONFIG_DIR # 3
cp .../priv_validator_key.json   $CONFIG_DIR # 4  <- the file that panics when absent

So an init interrupted between (2) and (4) leaves the guard file present and the keys missing. Every subsequent start takes the Already initialized branch, copies nothing, and panics.

Unrecoverable by restarting — the node wedges permanently. aristotle reached that state via the repeated build failures earlier today (see #61, #63).

This is latent for every 0g node. galileo escaped only because its init happened to complete uninterrupted.

Fix

Guard on everything 0gchaind needs to boot — both config keys and the data state file — instead of one proxy file on the wrong volume.
Write the keys first and the guard file last, so an interrupted init re-runs on the next start rather than latching.

Tested locally against the real binary

Not reasoned about — reproduced. aristotle-v1.0.6's actual 0gchaind (version → e8e1071) runs on the controller, so the matrix ran the real 0gchaind init with a sandboxed HOME_DIR:

scenario	before	after
fresh volumes	keys created	keys created — no regression
wedged (state, no keys)	keys MISSING, `Already initialized`	keys created, `identity incomplete → initializing`
partial (state + node_key only)	keys MISSING	keys created
healthy (all present)	no re-init	no re-init

The wedged row reproduces aristotle's production panic exactly, and the fix clears it.

Non-destructiveness was tested explicitly rather than assumed: in the healthy case a SENTINEL written into priv_validator_key.json survives the run, so the fix can never clobber an already-initialized node. sh -n clean; the geth branch is untouched.

Caveat worth stating

On an affected node this regenerates node_key.json and priv_validator_key.json. On a validator that would matter — these are RPC nodes that never sign, so a fresh identity is harmless, and chain data on the _zerog volume is untouched.

After merge

aristotle needs a force-recreate on de-32 to re-run init and populate the keys. galileo is unaffected and currently syncing normally (height ~7.1k of 49.07M).

Related: #61 (download fails loudly), #63 (extraction fails loudly). This one is the same theme a step later — a guard that silently skipped the work it was guarding.

🤖 Generated with Claude Code

**ASSET-only** — `zero-gravity/scripts/init.sh`. Script-only, so no `./update.sh` and no gitlink change. ## Symptom aristotle @ rpc-de-32 crash-loops. `show-status` reports it as `syncing`, which is wrong — the CL never starts: ``` priv_validator_state.json found in /root/.0g. Continuing! Already initialized, continuing! panic: error calling provider ProvideBlsSigner (node-core/components/signer.go:46): key file does not exist at path: /root/.0g/config/priv_validator_key.json ``` The EL is a casualty, not the cause — it sits at `peercount=0` with zero engine traffic because nothing is driving it. ## Root cause: cross-volume inference, plus a write order that latches the failure `$DATA_DIR` and `$CONFIG_DIR` are **separate docker volumes** (`<node>_zerog` → `/root/.0g/data`, `<node>_config` → `/root/.0g/config`). The guard tested only: ```sh if [ ! -f "$DATA_DIR/priv_validator_state.json" ]; then ``` — inferring the state of the **config** volume from a file on the **data** volume. And the copies wrote that guard file *before* the keys: ```sh cp .../priv_validator_state.json $DATA_DIR # 2 <- guard file cp .../node_key.json $CONFIG_DIR # 3 cp .../priv_validator_key.json $CONFIG_DIR # 4 <- the file that panics when absent ``` So an init interrupted between (2) and (4) leaves the guard file present and the keys missing. Every subsequent start takes the `Already initialized` branch, **copies nothing**, and panics. **Unrecoverable by restarting — the node wedges permanently.** aristotle reached that state via the repeated build failures earlier today (see #61, #63). This is latent for **every** 0g node. galileo escaped only because its init happened to complete uninterrupted. ## Fix 1. **Guard on everything `0gchaind` needs to boot** — both config keys *and* the data state file — instead of one proxy file on the wrong volume. 2. **Write the keys first and the guard file last**, so an interrupted init re-runs on the next start rather than latching. ## Tested locally against the real binary Not reasoned about — reproduced. `aristotle-v1.0.6`'s actual `0gchaind` (`version` → `e8e1071`) runs on the controller, so the matrix ran the real `0gchaind init` with a sandboxed `HOME_DIR`: | scenario | before | after | |---|---|---| | fresh volumes | keys created | keys created — **no regression** | | **wedged** (state, no keys) | **keys MISSING**, `Already initialized` | **keys created**, `identity incomplete → initializing` | | partial (state + node_key only) | keys MISSING | keys created | | healthy (all present) | no re-init | no re-init | The **wedged** row reproduces aristotle's production panic exactly, and the fix clears it. Non-destructiveness was tested explicitly rather than assumed: in the healthy case a `SENTINEL` written into `priv_validator_key.json` **survives** the run, so the fix can never clobber an already-initialized node. `sh -n` clean; the `geth` branch is untouched. ## Caveat worth stating On an affected node this regenerates `node_key.json` and `priv_validator_key.json`. On a **validator** that would matter — these are RPC nodes that never sign, so a fresh identity is harmless, and chain data on the `_zerog` volume is untouched. ## After merge aristotle needs a `force-recreate` on de-32 to re-run init and populate the keys. galileo is unaffected and currently syncing normally (height ~7.1k of 49.07M). Related: **#61** (download fails loudly), **#63** (extraction fails loudly). This one is the same theme a step later — a guard that silently skipped the work it was guarding. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude added 1 commit 2026-08-12 19:34:15 +00:00

zero-gravity: init guard must check the CONFIG volume, not a data-volume proxy 92cdb31d17

aristotle @ rpc-de-32 crash-looped after its build was fixed:

    priv_validator_state.json found in /root/.0g. Continuing!
    Already initialized, continuing!
    panic: ProvideBlsSigner (node-core/components/signer.go:46):
           key file does not exist at path: /root/.0g/config/priv_validator_key.json

$DATA_DIR and $CONFIG_DIR are SEPARATE docker volumes (<node>_zerog and
<node>_config), but the guard tested only $DATA_DIR/priv_validator_state.json —
inferring the state of the config volume from a file on the data volume. Worse, the
copy order wrote that guard file BEFORE the two key files, so an init interrupted
between them left the data volume with the guard and the config volume without the
keys. Every later start then took the 'Already initialized' branch, copied nothing,
and panicked. Unrecoverable by restarting: the node wedges permanently. aristotle
got there via the repeated build failures earlier today.

Fix, two parts:
  - guard on everything 0gchaind needs to boot (both config keys AND the data state
    file), not one proxy file on the wrong volume;
  - write the keys FIRST and the guard file LAST, so an interrupted init re-runs on
    the next start instead of latching into the wedged state.

Tested locally against the real 0gchaind binary (aristotle v1.0.6), sandboxed HOME:

  scenario                     before            after
  fresh volumes                keys created      keys created        (no regression)
  wedged (state, no keys)      keys MISSING      keys created        (unwedges)
  partial (state+node_key)     keys MISSING      keys created
  healthy (all present)        no re-init        no re-init, and a SENTINEL written
                                                 into priv_validator_key.json survives
                                                 -> never clobbers an initialized node

The 'wedged' row reproduces aristotle's production failure exactly.

Note this regenerates node identity on an affected node (node_key.json,
priv_validator_key.json). These are RPC nodes that never sign, so a fresh identity is
harmless; chain data on the _zerog volume is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude manually merged commit 92cdb31d17 into main

2026-08-12 19:37:52 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: StakeSquid/ethereum-rpc-docker#64