A hung check-health.sh (aztec-testnet, looping on an unresponsive reference RPC)
blocked show-status.sh's parallel 'wait' for 3.5h, hanging the whole fleet
rpc-update and holding the deploy lock. Each curl was bounded (-m 3) and the
retry loop capped (3x), but the call itself wasn't time-bounded.
- sync-status.sh: wrap each check-health.sh call in 'timeout ${HC_TIMEOUT:-30}'
(-> exit 124 + 'timeout' status on overrun).
- show-status.sh: wrap the whole per-node sync-status.sh call in
'timeout ${SYNC_TIMEOUT:-60}' so the parallel wait can never block forever.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- BUG 1: Add error checking after tar extraction for both LOCAL and REMOTE-CACHE branches
- Check exit status of tar -I zstd -xf commands
- Print error to stderr and exit non-zero on failure
- Prevents silent truncation where corrupt/incomplete backup extracts partial data
- Mirrors existing remote-STREAM branch error handling
- BUG 2: Fix REMOTE branch to resume incomplete aria2c downloads
- Check for presence of <file>.aria2 control file as incomplete signal
- aria2c -c continues/resumes download when .aria2 file exists
- Only skip download when file exists AND no .aria2 control file remains
- aria2 deletes .aria2 sidecar on successful completion, making it a reliable signal
- Maintain all existing flags: aria2c -c -Z -x8 -j8 -s8 -d
- Preserve reth guard logic and static-file offload behavior unchanged
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add RETH GUARD to clone-backup.sh and clone-node.sh: when the config name
contains 'reth', skip the whole /slowdisk static-file symlink offload and
extract everything onto the primary disk (equivalent to --no-slowdisk).
This matches the already-correct restore-volumes.sh behavior.
Reason: reth refuses to start when its static_files directory is a symlink,
failing at boot with 'failed to create dir static_files: File exists'.
- Add --keep-directory-symlink to all tar extraction options in both scripts
for the SLOWDISK path. This allows tar to extract files THROUGH the
pre-created directory symlinks instead of trying to mkdir over them
(which fails with 'Cannot mkdir: File exists'). This matches the
already-correct restore-volumes.sh behavior.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Rename doma testnet relay from celestia to op-alt (ghcr.io/celestiaorg/op-alt-da)
- Add op-alt relay for doma mainnet with image tag 0.15.0 (not v0.15.0)
- Vendor complete mainnet rollup.json with alt_da (GenericCommitment)
- Fix per-network relay settings: namespace + CELESTIA_*_RPC env vars
- Switch mainnet op-geth to use vendored rollup.json (bind-mounted)
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add relay template (templates/relays/celestia.doma.yml) for op-alt-da
- Add celestia relay config to op/doma stack in context.yml
- Add relay: celestia to op/doma testnet in config.yml
- Add op-node.doma.yml template with ALTDA env vars gated on relay_name
- Update op/doma/testnet/rollup.json with complete config including alt_da block
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Conduit's testnet rollup.json (doma-dev-ix58nm4rnd) omits chain_op_config,
crashing op-node v1.19.0 with 'cfg.Rollup.ChainOpConfig is nil'. Vendor a
complete rollup.json (chain_op_config eip1559 6/50/250, confirmed identical
in mainnet rollup.json and testnet genesis optimism config) plus the
post-fjord fork times baked in, mounted via custom_config so op-node uses
the local file instead of curling Conduit's incomplete one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Doma testnet (Conduit slug doma-dev-ix58nm4rnd) on Conduit's purpose-built
conduit-op-reth image - better-tested for Conduit chains than upstream op-geth.
Fork times granite/holocene/isthmus/jovian (missing from Conduit's bare
rollup.json) supplied via env/op/doma/testnet/node.env. Experiment to validate
a clean canonical sync before redoing mainnet on op-reth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conduit's bare rollup.json for doma-mainnet omits jovian_time entirely, but the
EL genesis activates Jovian at 1769065201. Without the override op-node has no
Jovian fork time and mis-derives at the boundary (chain forks off canonical).
Same fix pattern as bob-sepolia. Override comes from env/op/doma/mainnet/node.env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bepolia activated the Fusaka/Fulu hard fork at 2026-05-27 16:00 UTC (unix
1779897600), which ships new EVM-inflation-withdrawal values. beacon-kit
v1.3.4 (pre-fork) crash-loops replaying the fork-boundary block 20513284:
'first withdrawal is not the EVM inflation withdrawal' (RestartCount 3742 on
us-41). The EL (bera-reth v1.4.0) already speaks the Fusaka engine API
(newPayloadV4P11 etc.) and accepted the post-fork block, so only the CL is
behind. v1.4.0-rc3 sets the Fulu fork time + new inflation values. Applies
to all berachain chains' beacon-kit default; only bepolia is deployed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The container ran as user 0:994 and accessed the docker socket via group
membership, but the host docker group GID is auto-assigned and varies per
host (e.g. uk-8 is 988, not 994), so the hardcoded gid silently breaks
telegraf's docker input wherever it differs (uk-8 was in a restart loop:
permission denied on /var/run/docker.sock). Run as root (0:0) with
entrypoint [telegraf] to skip the image's gosu privilege-drop, so telegraf
reads the socket as its owner regardless of the host docker gid. Works
uniformly fleet-wide; no regression on hosts where the gid happened to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
op-geth was stuck at genesis (block 0, 0 L2 peers for 31h) because execution-layer
snap-sync had no peers. node_sync=true -> OP_NODE_SYNCMODE=consensus-layer (derive
L2 from L1) + geth --syncmode=full. Diagnosed by cursor on de-13.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
netdev (netlink) was reading the container's own veth (idle) -> node_network_*
showed ~0 on every host. Host network lets it see real host interfaces, so
bandwidth (egress vs tariff quota) works. Removed networks/expose (exclusive
with network_mode:host).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds diskIO to cadvisor --enable_metrics so per-container disk read/write
(container_fs_reads/writes_bytes_total) is exposed — lets node attribution
name which node is the noisy IO neighbor, not just flag the host.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
prometheus-scrape.metrics_path was '/' which made prometheus-docker-sd scrape
the HTML root and fail with 'INVALID is not a valid start token', leaving the
targets up=0. Fixes per-container (cadvisor) + host (node_exporter) metrics so
they can be wired into the DRPC insights MCP for per-node resource attribution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mint a Viewer Grafana service-account token on a host and print it to stdout,
so each box's local Grafana (dshackle + node_exporter + cadvisor) can be wired
into the DRPC insights MCP (grafana_servers.yaml). Idempotent SA, fresh token
per call.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the dRPC gateway's per-chain "how many blocks behind is ok" model instead of a
fixed 2s/5s timestamp tolerance:
- check-health.sh: compare the reference head vs local head by BLOCK NUMBER and classify
with the chain's dRPC lag thresholds (LAGGING_LAG/SYNCING_LAG, in blocks, from
chains.yaml). dRPC uses the two thresholds inconsistently across chains (sometimes
lagging<syncing, sometimes the reverse) so the smaller is the online boundary and the
larger the syncing/drop boundary. Defaults 2/6 when a chain has no thresholds.
- multicurl.sh: also skip responses with result:null (a lagging endpoint lacking the
requested block) so the fallback reference URLs are actually tried. Previously the first
endpoint's {"result":null} was accepted as success -> fallbacks never ran, and the null
reference hash made check-health report false "forked" (the online/forked flapping).
- sync-status.sh: resolve the lag thresholds (by drpc slug or chain id) and export
LAGGING_LAG/SYNCING_LAG.
- reference-rpc-endpoint.sh: add --lags and --block-time-ms lookups.
- reference-rpc-endpoint.json: regenerated with per-chain block_time_ms + lagging_lag +
syncing_lag (additive).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Revert to the prune-cycle model for minimal nodes: the normal compose serves
RPC with no --init.prune, and a generated .prune.yml runs --init.prune=minimal,
driven periodically by prune-if-prunable (same mechanism as pruned/full). Minimal
nodes are seeded from a pruned backup, then pruned to minimal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Minimal BSC nodes lack the state to serve eth_getProof and return
"header not found" (especially while catching up after a restart). The
drpc gateway probes eth_getProof and marks the WHOLE upstream unavailable
on failure, so e.g. bsc-mainnet-bsc-minimal on us-32 showed unavailable in
US-East despite the node being online and at head.
Disable eth_getProof on bsc minimal upstreams only (network=bsc,
db_type=minimal) so they stay available for every other method. Archive/
pruned bsc and all other nodes keep serving getProof.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
X Layer mainnet/testnet op-reth now use the OKX xlayer/op-reth:v0.0.4.1 fork
(generic op-reth can't read the X Layer DB format), the image's built-in
xlayer-<network> chain spec, --rpc.legacy-url for post-snapshot history gap-fill,
and the OKX archive-compatible reth.toml (light pruning only: merkle_changesets
distance=10064). Switched the reth profile from full_trace to archive_trace and
deleted the pruned variants — a "pruned" compose over archive snapshot data
crash-loops on a block-height mismatch. Requires the official OKX reth snapshot
pre-loaded into the volume; do NOT sync from scratch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct the minimal-node model: the minimal compose itself carries
--init.prune=minimal (prunes to genesis+head on start) instead of relying on a
separate .prune.yml — otherwise a minimal node was byte-identical to pruned in
normal operation and had no way to enforce minimal state. Removed the redundant
minimal .prune.yml files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
op-node was following the sequencer tip in execution-layer mode while op-reth stayed
at genesis (blocks buffered, never canonical). Switch to consensus-layer sync so
op-node drives L1 derivation, add op-reth --disable-discovery, and use debug_geth
(eth_getBlockReceipts) for the L1 RPC kind — "basic" per-tx receipts hit "receipt 0
has unexpected nil block number" on the Sepolia L1 endpoint. OP_NODE_OVERRIDE_JOVIAN
=1772548201 is belt-and-suspenders (jovian_time already in rollup.json). Applies to
all bob-sepolia EL variants. Regenerated from the vibe-node generator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
op-reth/op-geth sepolia now run op-node consensus-layer sync (empty DB is never
P2P-backfilled) with --disable-discovery, derive L1 via debug_geth against an
operator-set full-history Sepolia RPC (SUPERSEED_SEPOLIA_L1_EXECUTION_RPC, falling
back to ETHEREUM_SEPOLIA_EXECUTION_RPC), and serve a genesis.json with the invalid
top-level excessBlobGas/blobGasUsed nulls removed (eip1559DenominatorCanyon=250
retained). Regenerated from the vibe-node generator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per cursor cosmos handoff. (1) Traefik WS router rule Headers(Upgrade,websocket) is
case-SENSITIVE -> clients sending 'Upgrade: WebSocket' (python websocket-client) fall
through to the RPC router (200/400 not 101). rpc-client.yml now emits
HeadersRegexp(Upgrade,(?i)websocket) for all split-WS chains; regenerated cosmos+avalanche.
(2) Refactor into cometbft-common.sh: ct_configure_statesync skips if data/application.db
exists; new ct_ensure_wasm seeds CosmWasm/IBC-08 wasm (statesync omits them). init.sh calls
both. README documents wasm/version/WS gotchas. 45 other split-WS composes get HeadersRegexp
on their next regen.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Our L1 geth (e.g. rpc-de-31 ethereum-mainnet-geth) returns null for the
per-tx receipt path on historical blocks but serves eth_getBlockReceipts
fine. op-node's 'basic' kind only uses the per-tx batch, so chains
deriving from genesis (e.g. Katana) hit the broken path immediately;
chains already synced past it don't. 'standard' uses eth_getBlockReceipts
and keeps the per-tx batch as fallback (strict superset of 'basic'), so
it is safe for every OP-stack node.
Flips OP_NODE_L1_RPC_KIND default across all 176 op-node compose files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dshackle's WS head subscription hit the RPC root (got HTTP 200, not a WS upgrade).
Set client_ws_path=/websocket -> rpc-client.yml emits the avalanche-style priority-100
WS router (Header Upgrade:websocket -> replacepath /websocket) so WS reaches CometBFT's
/websocket. RPC router unchanged (stripprefix -> root).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New minimal profile for arbitrum one/nova/sepolia: a pruned-style node whose
prune cycle uses --init.prune=minimal (most aggressive: genesis+head only)
instead of full. Separate composes so it can be tested independently of the
production pruned nodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The polkachu seed was refusing -> 0 peers -> statesync couldn't discover snapshots.
Use client_staticpeers (cosmos-hub is a CLIENT in our stack) with 8 live peers from
polkachu /net_info. Template now reads client_staticpeers/client_bootnodes (standard attrs).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CometBFT config.toml uses underscore keys; the sed targeted hyphens (rpc-servers etc.)
so they never matched -> rpc_servers stayed empty -> 'at least 2 RPC servers required,
got 0'. Now section-anchored to [statesync] + [_-] tolerant. (This is also why sei
never held chainhead — same hyphen bug in its bespoke init.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes prune-version drift that corrupted nodes (prune ran an older nitro binary
than the node). All .prune.yml now regenerate from the same config as the normal
compose via client_needs_prune.
- arb nitro: nova/one/sepolia .prune.yml bumped to the node's version (v3.10.1)
- avalanche: add .prune.yml variants that mount a /config/prune chain-config with
offline-pruning-enabled, plus avalanche/{mainnet,fuji}/prune/C/config.json
- archive profiles are excluded (db_type==pruned gate) so they are never pruned
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CometBFT RPC :26657 (the dshackle upstream) is served via rpc.laddr regardless;
minimal 'gaiad start --home --minimum-gas-prices' is the valid invocation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ct_configure_statesync read .block.header.height (null — CometBFT wraps in .result) so
statesync silently skipped -> gaiad fell back to genesis replay (panic). Use
'.result.X // .X' (robust to wrapped/unwrapped). Same fallback in check-health --cosmos.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
