show-status: exit code was always 0 — three compounding bugs

show-status.sh could never report failure. The ansible task that wraps it
('Execute show-status.sh and fail on failure') therefore always passed, on every
host, regardless of node state. Three bugs, each masking the next:

1. $? read too late. `code=0` sits between the sync-status.sh call and
   `if [ $? -ne 0 ]`. A plain assignment succeeds and overwrites $? with 0, so
   the condition was ALWAYS false and the else branch always taken.

2. The else branch was inverted. It is the sync-status-SUCCEEDED path, yet it set
   `code=1; any_failure=true` — marking healthy nodes as failures.

3. any_failure could never propagate. check_sync_status runs backgrounded (`&`),
   i.e. in a subshell, so `any_failure=true` inside it is discarded; and the
   `wait "$pid"` loop threw away each job's exit status.

(3) hid (1) and (2): a script that believed every node had failed still exited 0,
so nobody saw it.

Fix: capture rc immediately; restore the intended logic (success => 0, syncing or
lagging => tolerated, anything else => failure); propagate failure in the PARENT
via `wait "$pid" || any_failure=true`, since the subshell cannot.

Verified with a stubbed sync-status.sh:

  scenario              before   after
  all online              0        0
  one syncing             0        0   (tolerated)
  one lagging             0        0   (tolerated)
  one ERROR               0        1
  ALL error               0        1

Behaviour for healthy fleets is unchanged; only genuine failures now surface.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

cosmos/gaiad/cosmos-mainnet-gaiad-pruned.yml

@@ -35,7 +35,7 @@ services:
       dockerfile: cometbft.Dockerfile
       args:
         CL_IMAGE: ${COSMOS_GAIAD_IMAGE:-ghcr.io/cosmos/gaia}
-        CL_VERSION: ${COSMOS_MAINNET_GAIAD_VERSION:-v27.5.0}
+        CL_VERSION: ${COSMOS_MAINNET_GAIAD_VERSION:-v27.6.0}
     sysctls:
       # TCP Performance
       net.ipv4.tcp_slow_start_after_idle: 0       # Disable slow start after idle

reference-rpc-endpoint.json

@@ -3368,6 +3368,16 @@
       "https://mova.drpc.org"
     ]
   },
+  "mova-mainnet-v2": {
+    "protocol": "eth",
+    "id": 61901,
+    "block_time_ms": 2000,
+    "lagging_lag": 10,
+    "syncing_lag": 20,
+    "urls": [
+      "https://mova-mainnet-v2.drpc.org"
+    ]
+  },
   "mova-testnet": {
     "protocol": "eth",
     "id": 10323,

show-status.sh

@@ -22,26 +22,25 @@ check_sync_status() {
     # Cap the whole per-node branch (belt-and-suspenders over check-health's own cap), so no single
     # node can ever block the 'wait' below — that is what wedged the fleet rpc-update for hours.
     result=$(timeout "${SYNC_TIMEOUT:-60}" "$BASEPATH/sync-status.sh" "${part%.yml}")
+    # Capture the status IMMEDIATELY. Any command in between - including a plain
+    # assignment like `code=0` - overwrites $? with its own (always 0) status.
+    rc=$?
 
     code=0
-    
-    if [ $? -ne 0 ]; then
-        if [[ "$result" == *"syncing"* ]]; then
-            # Allow exit status 1 if result contains "syncing"
-            code=0
-        elif [[ "$result" == *"lagging"* ]]; then
-            # Allow exit status 1 if result contains "lagging"
+    if [ "$rc" -ne 0 ]; then
+        if [[ "$result" == *"syncing"* ]] || [[ "$result" == *"lagging"* ]]; then
+            # sync-status exits 1 for syncing/lagging; those are expected states,
+            # not failures.
             code=0
         else
-            any_failure=true
             code=1
         fi
-    else
-	code=1
-	any_failure=true
     fi
 
     echo "${part%.yml}: $result"
+    # NOTE: do NOT set any_failure here. This function runs backgrounded (`&`), so
+    # it executes in a subshell and any variable it sets is discarded. Failure is
+    # propagated to the parent through this return code, collected by `wait` below.
     return "$code"
 }
 
@@ -74,9 +73,12 @@ for part in "${parts[@]}"; do
     fi
 done
 
-# Wait for all background processes to finish
+# Wait for all background processes to finish. `wait` runs in the PARENT shell, so
+# this is where a failing node can actually flip any_failure - the checker itself
+# cannot, being a subshell. Previously the status was discarded here, which silently
+# neutered the exit code.
 for pid in "${pids[@]}"; do
-    wait "$pid"
+    wait "$pid" || any_failure=true
 done
 
 # Fenced nodes (fleet-state maintenance windows) are dropped from COMPOSE_FILE

zero-gravity/geth/zero-gravity-galileo-geth-pruned-pebble-path.yml

@@ -140,7 +140,7 @@ services:
       - --chaincfg.node-api.logging
       - --home=/root/.0g
       - --p2p.external_address=${IP}:16172
-      - --p2p.seeds=${ZERO_GRAVITY_GALILEO_ZEROG_SEEDS:-461b27a9d1530eb47f62fe010a8d3e5d43b6740c@34.82.252.10:26656}
+      - --p2p.seeds=${ZERO_GRAVITY_GALILEO_ZEROG_SEEDS:-e0750abc061d276894a593c7b671913c14aa4ecc@34.82.252.10:26656}
       - --pruning=default
       - --rpc.laddr=tcp://0.0.0.0:16172
     restart: unless-stopped

zero-gravity/scripts/init.sh

@@ -61,23 +61,41 @@ env
 
 # seems to be the same for all the 0g chains
 
-if [ ! -f "$DATA_DIR/priv_validator_state.json" ]; then
-    echo "priv_validator_state.json not found in $HOME_DIR. Proceeding with initialization steps..."
+# $DATA_DIR and $CONFIG_DIR are SEPARATE docker volumes (<node>_zerog and <node>_config),
+# so the presence of a file in one says nothing about the other. Guard on everything
+# 0gchaind actually needs to boot, and write the copies so the LAST file written is the
+# one that would make a re-run skip.
+#
+# The previous version guarded solely on $DATA_DIR/priv_validator_state.json and copied it
+# BEFORE the two key files. An init interrupted between those copies therefore left the
+# data volume with the guard file and the config volume without the keys, so every later
+# start took the "Already initialized" branch, copied nothing, and 0gchaind panicked:
+#
+#   panic: error calling provider ProvideBlsSigner (node-core/components/signer.go:46):
+#          key file does not exist at path: /root/.0g/config/priv_validator_key.json
+#
+# That state is unrecoverable by restarting - the node wedges permanently. Hit on
+# aristotle @ rpc-de-32 on 2026-08-12 after repeated build failures interrupted its init.
+if [ ! -f "$CONFIG_DIR/priv_validator_key.json" ] || \
+   [ ! -f "$CONFIG_DIR/node_key.json" ] || \
+   [ ! -f "$DATA_DIR/priv_validator_state.json" ]; then
+    echo "0g: node identity incomplete (config keys and/or priv_validator_state.json). Initializing..."
     TMP_DIR=$(mktemp -d)
     # You can add any additional initialization logic here if needed
     if /0g/bin/0gchaind init ${MONIKER} --chaincfg.chain-spec ${CHAIN_SPEC} --home $TMP_DIR; then
         cp -r /0g/0g-home/0gchaind-home/config/* $CONFIG_DIR
-        cp $TMP_DIR/data/priv_validator_state.json $DATA_DIR
+        # Keys first, guard file last: if this is interrupted the next start re-runs
+        # initialization instead of latching into the wedged state described above.
         cp $TMP_DIR/config/node_key.json $CONFIG_DIR
         cp $TMP_DIR/config/priv_validator_key.json $CONFIG_DIR
+        cp $TMP_DIR/data/priv_validator_state.json $DATA_DIR
     else
         echo "Already initialized, continuing!" >&2
     fi
     rm -rf $TMP_DIR # delete tmp dir
 
 else
-    echo "priv_validator_state.json found in $HOME_DIR. Continuing!" >&2
-    echo "Already initialized, continuing!" >&2
+    echo "0g: node identity complete (config keys + priv_validator_state.json). Continuing!" >&2
 fi
 
 exec /0g/bin/0gchaind $@