Compare commits

..

5 Commits

Author SHA1 Message Date
cf91b6f14d Merge pull request 'linea: reduce Maru engine API parallelism' (#49) from issue-2412 into main 2026-08-28 05:25:30 +00:00
529a91f0d7 Merge pull request 'show-status: exit code was always 0 — three compounding bugs' (#66) from fix-show-status-exit-code into main 2026-08-28 05:25:11 +00:00
502a75b2fd show-status: exit code was always 0 — three compounding bugs
show-status.sh could never report failure. The ansible task that wraps it
('Execute show-status.sh and fail on failure') therefore always passed, on every
host, regardless of node state. Three bugs, each masking the next:

1. $? read too late. `code=0` sits between the sync-status.sh call and
   `if [ $? -ne 0 ]`. A plain assignment succeeds and overwrites $? with 0, so
   the condition was ALWAYS false and the else branch always taken.

2. The else branch was inverted. It is the sync-status-SUCCEEDED path, yet it set
   `code=1; any_failure=true` — marking healthy nodes as failures.

3. any_failure could never propagate. check_sync_status runs backgrounded (`&`),
   i.e. in a subshell, so `any_failure=true` inside it is discarded; and the
   `wait "$pid"` loop threw away each job's exit status.

(3) hid (1) and (2): a script that believed every node had failed still exited 0,
so nobody saw it.

Fix: capture rc immediately; restore the intended logic (success => 0, syncing or
lagging => tolerated, anything else => failure); propagate failure in the PARENT
via `wait "$pid" || any_failure=true`, since the subshell cannot.

Verified with a stubbed sync-status.sh:

  scenario              before   after
  all online              0        0
  one syncing             0        0   (tolerated)
  one lagging             0        0   (tolerated)
  one ERROR               0        1
  ALL error               0        1

Behaviour for healthy fleets is unchanged; only genuine failures now surface.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 19:56:10 +00:00
80b0e4253e linea/mainnet/maru: reduce blocks-parallelism 10→2 and blocks-batch-size 10→5 to reduce engine API saturation 2026-08-02 18:03:02 +00:00
c5f9e5000f linea: lower desync-tolerance from 100000 to 64 to allow range-sync mode 2026-08-02 14:51:49 +00:00
5 changed files with 20 additions and 19 deletions

View File

@@ -49,12 +49,12 @@ port = 8080
peer-chain-height-polling-interval = "5s"
el-sync-status-refresh-interval = "5s"
sync-target-selection = "Highest"
desync-tolerance = 100000
desync-tolerance = 64
[syncing.download]
block-range-request-timeout = "10s"
blocks-batch-size = 10
blocks-parallelism = 10
blocks-batch-size = 5
blocks-parallelism = 2
max-retries = 5
backoff-delay = "1s"
use-unconditional-random-download-peer = false

View File

@@ -30,7 +30,7 @@ x-logging-defaults: &logging-defaults
services:
zircuit-garfield-op-reth-pruned:
image: ${ZIRCUIT_RETH_IMAGE:-ghcr.io/conduitxyz/conduit-op-reth}:${ZIRCUIT_GARFIELD_RETH_VERSION:-v2.1.1}
image: ${ZIRCUIT_RETH_IMAGE:-ghcr.io/conduitxyz/conduit-op-reth}:${ZIRCUIT_GARFIELD_RETH_VERSION:-v2.1.2}
sysctls:
# TCP Performance
net.ipv4.tcp_slow_start_after_idle: 0 # Disable slow start after idle

View File

@@ -30,7 +30,7 @@ x-logging-defaults: &logging-defaults
services:
zircuit-mainnet-op-reth-pruned:
image: ${ZIRCUIT_RETH_IMAGE:-ghcr.io/conduitxyz/conduit-op-reth}:${ZIRCUIT_MAINNET_RETH_VERSION:-v2.1.1}
image: ${ZIRCUIT_RETH_IMAGE:-ghcr.io/conduitxyz/conduit-op-reth}:${ZIRCUIT_MAINNET_RETH_VERSION:-v2.1.2}
sysctls:
# TCP Performance
net.ipv4.tcp_slow_start_after_idle: 0 # Disable slow start after idle

View File

@@ -3441,7 +3441,6 @@
"syncing_lag": 40,
"urls": [
"https://rpc.testnet.chain.robinhood.com/rpc",
"https://robinhood-sepolia-rpc.publicnode.com",
"https://robinhood-testnet.drpc.org"
]
},

View File

@@ -22,26 +22,25 @@ check_sync_status() {
# Cap the whole per-node branch (belt-and-suspenders over check-health's own cap), so no single
# node can ever block the 'wait' below — that is what wedged the fleet rpc-update for hours.
result=$(timeout "${SYNC_TIMEOUT:-60}" "$BASEPATH/sync-status.sh" "${part%.yml}")
# Capture the status IMMEDIATELY. Any command in between - including a plain
# assignment like `code=0` - overwrites $? with its own (always 0) status.
rc=$?
code=0
if [ $? -ne 0 ]; then
if [[ "$result" == *"syncing"* ]]; then
# Allow exit status 1 if result contains "syncing"
code=0
elif [[ "$result" == *"lagging"* ]]; then
# Allow exit status 1 if result contains "lagging"
if [ "$rc" -ne 0 ]; then
if [[ "$result" == *"syncing"* ]] || [[ "$result" == *"lagging"* ]]; then
# sync-status exits 1 for syncing/lagging; those are expected states,
# not failures.
code=0
else
any_failure=true
code=1
fi
else
code=1
any_failure=true
fi
echo "${part%.yml}: $result"
# NOTE: do NOT set any_failure here. This function runs backgrounded (`&`), so
# it executes in a subshell and any variable it sets is discarded. Failure is
# propagated to the parent through this return code, collected by `wait` below.
return "$code"
}
@@ -74,9 +73,12 @@ for part in "${parts[@]}"; do
fi
done
# Wait for all background processes to finish
# Wait for all background processes to finish. `wait` runs in the PARENT shell, so
# this is where a failing node can actually flip any_failure - the checker itself
# cannot, being a subshell. Previously the status was discarded here, which silently
# neutered the exit code.
for pid in "${pids[@]}"; do
wait "$pid"
wait "$pid" || any_failure=true
done
# Fenced nodes (fleet-state maintenance windows) are dropped from COMPOSE_FILE