The reload is in-process and takes 10-30 ms; the fixed sleep was the window in
which a removed upstream was already unrouted in dshackle while the gateway still
held the old announcements and kept sending queries it could no longer answer.
Poll the log for 'Reloading config has been completed' (or the fail/drop
markers), SETTLE stays the upper bound. Operator 2026-09-19.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KjqejYycVyVvrxV9aTDhgX
Regenerated from vibe-node context.yml (global nitro pin + robinhood, plume
mainnet/testnet overrides). robinhood archive profiles (not rendered, no host
runs them) bumped by hand for parity. alephzero stays on its own v3.5.3 pin.
Regen also picked up the pending arc reference-rpc-endpoint list.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KjqejYycVyVvrxV9aTDhgX
bor snapshot prune-block wants the directory holding the freezer tables
(bodies.*.cdat) - the modern layout splits ancient/ into chain/ + state/,
so ancient/ root fails with 'ancient chain segments already extracted' and
the prune container crash-loops (de-12, 2026-08-02). Matches BSC's
prune-block usage which targets .../ancient/chain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A 0G sidecar volume (ronin eigenda-proxy) FATAL'd the whole backup before the
manifest was written - verify found no new backup and the refresh pipeline's
bench-retire escalated 4 cycles in a row, head-of-line blocking the single
refresh lane. Sidecars are legitimately tiny; the <1G fail-closed floor now
applies only to the main data volume (key == compose basename).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HSK's published mainnet genesis sets isthmusTime but omits pragueTime; op-geth's
OP-Stack validation rejects it ('PragueTime (<nil>) must equal IsthmusTime'), so
geth init aborts and the EL runs on a stale chainconfig. testnet genesis correctly
sets pragueTime=isthmusTime. Adding it here makes init pass. (Runtime fork activation
is handled by GETH_OVERRIDE_* in vibe-node geth.env; this fixes init + matches testnet.)
Co-Authored-By: Claude <noreply@anthropic.com>
-jwt volumes (EL/CL shared secret) are regenerated at create-node and never
backed up, same as -secrets. Two fixes:
- clone-backup.sh: extend the fd8dd15b secrets skip to *-jwt (both paths)
- list-restorable.sh: exclude *-jwt/*-secrets from the all-volumes-required
restorability check. Both plasma composes showed Restorable: 0 for days
(mainnet 407G + testnet 98G+29G data backups present since 07-18/19)
purely because rpc_plasma-*-jwt had no backup; this misled the 07-16
plasma restore assessment ('de-32: NO plasma backup').
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Secrets volumes are regenerated at create-node and generally have no
backup by design. Counting the skip as Failed (exit 1) false-failed
whole provisions whose data volumes transferred fine (tempo de-32->de-27,
job e9634e66: executor's broken-pipe guard reset the action and would
have re-cloned everything). Reported by claude@ (inbox 829bc624).
- no-backup on a *-secrets volume -> return 2 (skip), both nc and ssh paths
- rc=2 tracked as Skipped in the summary, excluded from the failure count
- exit status now keys on failed_volumes only
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The false default forced every op-node restart to walk back to the finalized head
before driving the EL. On nodes whose derivation lags days behind their serving
gossip head (hashkey us-40: 510k blocks / 12 days), that froze the EL for hours
while dRPC de-routed the sole US provider. true resumes at the unsafe head in
minutes; safe/finalized still verify and catch up in the background. ronin
already ran true per-chain (same lesson); xlayer keeps its deliberate false pin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the alpine genesis/rollup init containers and their
service_completed_successfully gate for all hashkeychain variants. The gate
wedged us-40: a reconcile compose-up recreated the op-reth EL but not the
init-gated op-node CL -> fresh EL + stale CL -> 2h engine freeze -> sole-provider
market -> $0 (2026-07-21). genesis.json now committed beside the existing
rollup.json so render_templates auto-detects custom_genesis/custom_rollup and
binds the local files; CL now depends only on the EL (service_started,
restart:true), identical to op-mainnet/op-sepolia. No S3 on the boot path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Catch up from v1.2.1/1.0.1 pins so mainnet and sepolia followers track the current Consensys/maru release.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add bootstrap-peers.sh and fall back to stakesquid.eu/bootstrap when local peerset capture is empty; optional --seed-bootstrap on connect-peers.
Co-authored-by: Cursor <cursoragent@cursor.com>
Re-label consensys/maru FROM tag into image OCI labels (upstream leaves
version/vcs-ref empty) and scrape /metrics on :9090 instead of the 404ing
/debug/metrics/prometheus path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Hosts like rpc-us-32 put /var/lib/docker (or volumes/) on another mount;
tar -C / then EXDEV on every var/lib/docker/volumes/... member. Canonicalize
the deepest static prefix (readlink -f /var/lib/docker/volumes) and append a
--transform after SLOWDISK_TRANSFORMS. Keying only on /var/lib/docker misses
a real docker root with volumes/ as the symlink. No-op on normal hosts.
Per-volume _data static-file offload stays on SLOWDISK_TRANSFORMS.
Co-authored-by: Cursor <cursoragent@cursor.com>
dshackle's SIGHUP reload silently fails for upstream REMOVALS (and method-set
changes, same remove+add path): it logs "Config is not reloaded, cause - ...",
updates the in-memory config but leaves runtime selectors stale (drift), and keeps
serving the old config. No runtime admin API exists to mutate upstreams, so a
container restart is the only way to actually apply such changes.
Replace the blind SIGHUP with: per-container SIGHUP, then check post-SIGHUP logs
for the failure marker ("Config is not reloaded, cause -") or a dropped concurrent
HUP ("Reloading is in progress"); on either, docker restart to load config fresh.
"Reloading config has been completed" is NOT success (finally-block) — deliberately
not treated as such. Exit 0 when config ends up applied (via SIGHUP or restart);
non-zero only if a needed restart fails or the container doesn't come back.
Layered on top of the override-weave (apply-dshackle-overrides.py) and the 1:1
invariant gate (validate-dshackle-1to1.py) — both run unchanged before the reload.
Co-Authored-By: Claude <noreply@anthropic.com>
The splits completeness floor is TIME-based (trust hours complete since shipper
connect) and blind to event-loss-by-rotation (2026-07-10: raw_events rotated
every 1-2min, cruncher lost ~85% of us-32 events, floor still called the hours
complete). Shipper now injects {"__shipstat__":1,"sess":..,"sent":N,"ts":..}
every 1000 lines or 30s through the SAME tcp pipe. Cruncher diffs shipper-sent
vs DB-ingested per gateway/hour -> a reconciliation-based floor that catches the
loss a clock cannot. sess=connection-start epoch so resets on reconnect are
detectable. Logic in a mounted .awk file (not inline) to dodge sh/awk quoting.
Verified on busybox:1.36 (systime/fflush OK). NOT deployed until claude@'s
cruncher filters __shipstat__ from event parsing - coordinating via SIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same duplicated listener code, same two latent bugs. Not exercised today
(de-32 target has SLOWDISK=False) but any clone to a SLOWDISK=True host
would have failed identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The slowdisk --transform args contain '|' and quotes; the ssh+screen+bash -c
stack strips one quoting level too many and splits the pipeline (broke the
katana retry within minutes of fccf2792). A heredoc-generated
/tmp/transfer_<key>.sh is parsed exactly once on the target. Applies to both
the screen and nohup listener branches; the direct-ssh fallback already had a
single shell layer and stays inline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tar cannot create files through a directory symlink whose target is on
another device: open() fails with 'Invalid cross-device link' even under
--keep-directory-symlink (reproduced, GNU tar 1.34 and 1.35). This killed
the katana->uk-4 provision (place-provision-katana-EU-NorthWest-p1) and
means the slowdisk EXTRACT path never worked on hosts with a separate
/slowdisk device. Rewrite member paths with --transform so tar writes
directly into /slowdisk/rpc_<key>__data_<rel>; the symlink stays for the
container's runtime view. Also aligns restore-volumes' nested-path target
naming to the double-underscore convention clone-backup already uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified from crypto-org-chain/cronos-testnets: genesis, 3 official seeds,
t3 statesync RPC, basetcro denom; official daily snapshots exist for restore
(sidesteps the cosmos binary-upgrade gauntlet). Closes the planner's
'cronos-testnet: no known node_path' escalation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/root/rpc-local/dshackle-overrides.yaml (host-local, like rpc-local/compose.d) is
re-woven into main_configs/*.yaml by reload_dshackle.sh on every config push.
Method-disable decisions are local to each gateway's market (operator eviction
rule: cost_ratio = time_share/call_share at flat pay); this makes them durable
inputs instead of fragile output patches. Step 1 of moving dshackle config
generation onto the gateway hosts (step 2: ssh forced-command upstreams fetch,
generic dshackle.yaml include list derived from drpc-gateways.json).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each gateway serves its own regional market, so the eviction economics
(cost_ratio = time_share/call_share at flat per-call pay) are computed where
they apply: against the local dshackle's histograms. Silent on non-gateway
hosts. Offenders = candidates for per-upstream method-groups/methods disables
(operator eviction rule 2026-07-10). sendRawTransaction et al [protected].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All defaults preserve current effective behavior - this deploy is a rendering no-op
until a host_var overrides a value. Part of the resource-optimizer knob rollout
(resource-optimizer/knob-manifest.yaml is the registry; verify_manifest.py enforces
default sync between templates and envfile).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dshackle 0.79.10's beacon health validator (GET /eth/v1/node/health via
GenericSingleCallValidator) times out after 5000ms on the spec-compliant
empty-body 200 that nimbus returns, marking the upstream UNAVAILABLE in a
flap loop; live requests routed during flap windows fail 'Source was empty'.
This is the root cause behind dRPC's 617k failed validators POSTs complaint
(2026-07-09). Reproduced + workaround verified on an isolated dshackle
instance on de-14. Head tracking still governs availability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decoupled by design: dshackle writes the file regardless; a dead listener
or slow link stalls only the busybox sidecar. Retry resumes at file end
(bounded loss); the listener-side prometheus reconciliation surfaces gaps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gateway request-splits pilot (earnings misattribution fix): the accessLog
stub in main_configs pointed at /logs which was never mounted. Host-side
dir lets the splits tailer aggregate per-caller response counts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A silent fallback to primary-disk extraction violates the caller's NVMe
capacity math now that the planner treats slowdisk as a separate pool.
When SLOWDISK=True and the manifest's static sizes (accurate, replacing
the whole-archive x2 estimate that false-refuses large restores) do not
fit above the floor, error out with the explicit remedy: re-run with
--no-slowdisk as the conscious override. Operator directive 2026-07-09.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live test on rpc-uk-4 (unichain-mainnet pruned, 166G static_files moved to
/slowdisk + symlinked): reth healed consistency, initialized the
StaticFileProducer, and resumed committing canonical blocks - full
read/write through the symlink. The guard's premise (old reth refusing
symlinked static_files) no longer holds; pre-v2 reths fail loudly at start,
fallback is --no-slowdisk. Restores/clones of reth nodes to SLOWDISK=True
hosts now offload statics automatically (NVMe is the scarce resource).
Note: reth also has --datadir.static-files for a flag-based layout later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Candidate fix for the receipts RLP decode failure that blocks snap sync
past block 4,178,637 (2026-04-12): peers dropped with 'invalid message
(code 10) receipt 1: rlp: ListEnd not positioned at EOL' on the previous
newest published image.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fenced nodes are dropped from COMPOSE_FILE so show-status silently omitted
them - a fenced-but-running node looked 'gone'. rpc-update now writes
/root/rpc/.fenced (node | until | reason | owner per active window); print
it as a footer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 1MB near-empty guard (added for purged-volume zstd stubs) rejected every
legitimate _config volume archive (cronos: 68KB), so any node whose registry
entry includes a _config volume reported Restorable: 0 despite healthy
multi-GB data backups (cronos had three; 14 registry entries carry _config
volumes). _config archives now pass at >=4KB; data volumes keep the 1MB floor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hemi cannot full-sync across old BTC reorg points: EVM blocks referencing
orphaned Bitcoin Attributes ancestors are unexecutable once no peer retains
the stale BTC block (us-16 wedged at 3353463 on reorged-out block
00000000000000000001dcb3...b971). hemi's official l2-config.toml ships
SyncMode=snap. Registry: 150 entries gain node names (regen enrichment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Faithful copy of VIBE.md to seed cursor-specific tuning. The implement-vibe-node-change
wrapper, when run with VN_ENGINE=cursor (now the default), points cursor-agent at this
doc instead of VIBE.md so the two backends' instructions can diverge over time.
heimdall crash-looped on 'unknown flag: --rest-server.addr'. Per heimdalld start
--help, the bone-fork binary takes --rest-server (bool) to enable REST/LCD and
--laddr (default tcp://0.0.0.0:1317) for its listen address — there is no
--rest-server.addr. Swap it. (The rest_server-injection + idempotent-init fixes
from the prior PR worked: init now skips re-init, fetches the heimdall-109 genesis,
no more 'rest_server already defined'.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
heimdall crash-looped on 'toml: key rest_server is already defined'. init.sh
inserted 'rest_server = true' after [rpc] on EVERY boot (non-idempotent), piling up
duplicates; rest_server is not a valid CometBFT config.toml key anyway — REST is
enabled by the --rest-server flag in the start command.
- Remove the rest_server / rest_server_addr config injection.
- Self-heal: strip any previously-injected rest_server lines (so already-corrupted
volumes recover on next boot — no volume wipe needed).
- Guard heimdalld init to first boot only (config.toml absent) so restarts don't
reset node identity or re-parse the existing config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A parallel change reverted this to FROM shibaone/heimdall:v1.0.7-bone — an image
that exists on no registry, so the heimdall build fails. Restore the source build:
clone shibaone/heimdall@${CL_VERSION}, make install, then layer the CometBFT init
entrypoint. Pairs with vibe-node PR (node_build + standalone service name).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shibaone ships no docker images for the bone fork — only source + .deb config
packages — so both Dockerfiles must clone+build, not FROM a (nonexistent) image.
- bor.Dockerfile: clone shibaone/bor@${BOR_VERSION}, make bor, cp build/bin/bor
(was: alpine + wrong /src/build/bor path). golang:1.22.1 like upstream.
- cometbft.Dockerfile: clone shibaone/heimdall@${CL_VERSION}, make install
(was: FROM shibaone/heimdall:v1.0.7-bone — that tag does not exist on any
registry), then layer the CometBFT init entrypoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A hung check-health.sh (aztec-testnet, looping on an unresponsive reference RPC)
blocked show-status.sh's parallel 'wait' for 3.5h, hanging the whole fleet
rpc-update and holding the deploy lock. Each curl was bounded (-m 3) and the
retry loop capped (3x), but the call itself wasn't time-bounded.
- sync-status.sh: wrap each check-health.sh call in 'timeout ${HC_TIMEOUT:-30}'
(-> exit 124 + 'timeout' status on overrun).
- show-status.sh: wrap the whole per-node sync-status.sh call in
'timeout ${SYNC_TIMEOUT:-60}' so the parallel wait can never block forever.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- BUG 1: Add error checking after tar extraction for both LOCAL and REMOTE-CACHE branches
- Check exit status of tar -I zstd -xf commands
- Print error to stderr and exit non-zero on failure
- Prevents silent truncation where corrupt/incomplete backup extracts partial data
- Mirrors existing remote-STREAM branch error handling
- BUG 2: Fix REMOTE branch to resume incomplete aria2c downloads
- Check for presence of <file>.aria2 control file as incomplete signal
- aria2c -c continues/resumes download when .aria2 file exists
- Only skip download when file exists AND no .aria2 control file remains
- aria2 deletes .aria2 sidecar on successful completion, making it a reliable signal
- Maintain all existing flags: aria2c -c -Z -x8 -j8 -s8 -d
- Preserve reth guard logic and static-file offload behavior unchanged
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add RETH GUARD to clone-backup.sh and clone-node.sh: when the config name
contains 'reth', skip the whole /slowdisk static-file symlink offload and
extract everything onto the primary disk (equivalent to --no-slowdisk).
This matches the already-correct restore-volumes.sh behavior.
Reason: reth refuses to start when its static_files directory is a symlink,
failing at boot with 'failed to create dir static_files: File exists'.
- Add --keep-directory-symlink to all tar extraction options in both scripts
for the SLOWDISK path. This allows tar to extract files THROUGH the
pre-created directory symlinks instead of trying to mkdir over them
(which fails with 'Cannot mkdir: File exists'). This matches the
already-correct restore-volumes.sh behavior.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Rename doma testnet relay from celestia to op-alt (ghcr.io/celestiaorg/op-alt-da)
- Add op-alt relay for doma mainnet with image tag 0.15.0 (not v0.15.0)
- Vendor complete mainnet rollup.json with alt_da (GenericCommitment)
- Fix per-network relay settings: namespace + CELESTIA_*_RPC env vars
- Switch mainnet op-geth to use vendored rollup.json (bind-mounted)
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add relay template (templates/relays/celestia.doma.yml) for op-alt-da
- Add celestia relay config to op/doma stack in context.yml
- Add relay: celestia to op/doma testnet in config.yml
- Add op-node.doma.yml template with ALTDA env vars gated on relay_name
- Update op/doma/testnet/rollup.json with complete config including alt_da block
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Revert to the prune-cycle model for minimal nodes: the normal compose serves
RPC with no --init.prune, and a generated .prune.yml runs --init.prune=minimal,
driven periodically by prune-if-prunable (same mechanism as pruned/full). Minimal
nodes are seeded from a pruned backup, then pruned to minimal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct the minimal-node model: the minimal compose itself carries
--init.prune=minimal (prunes to genesis+head on start) instead of relying on a
separate .prune.yml — otherwise a minimal node was byte-identical to pruned in
normal operation and had no way to enforce minimal state. Removed the redundant
minimal .prune.yml files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New minimal profile for arbitrum one/nova/sepolia: a pruned-style node whose
prune cycle uses --init.prune=minimal (most aggressive: genesis+head only)
instead of full. Separate composes so it can be tested independently of the
production pruned nodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes prune-version drift that corrupted nodes (prune ran an older nitro binary
than the node). All .prune.yml now regenerate from the same config as the normal
compose via client_needs_prune.
- arb nitro: nova/one/sepolia .prune.yml bumped to the node's version (v3.10.1)
- avalanche: add .prune.yml variants that mount a /config/prune chain-config with
offline-pruning-enabled, plus avalanche/{mainnet,fuji}/prune/C/config.json
- archive profiles are excluded (db_type==pruned gate) so they are never pruned
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the restore-volumes.sh --no-slowdisk capability for live/backup clones.
Both scripts gate the target /slowdisk static-file offload on the target's
SLOWDISK env (case-insensitive, matches the Python-templated 'True') and accept
a --no-slowdisk flag that forces the offload off for one run. When SLOWDISK is
on but the target /slowdisk is too small for the static files, the clone warns
and aborts, telling the operator to re-run with --no-slowdisk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Use COMPOSE_FILE from .env to get list of nodes
- Match containers by service labels from compose files
- Output format now matches show-status (node paths, not container IDs)
- Sorted by RAM usage descending
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Path maps directly to compose file in subdirectories:
op/reth/base-mainnet-op-reth-archive-trace -> op/reth/base-mainnet-op-reth-archive-trace.yml
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix: handle full path like ethereum/geth/node-name (extract basename)
- Without args: show RAM per node (grouped by compose project), not per container
- Sorted by RAM usage descending
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Without argument: shows all containers sorted by RAM, total container RAM, server RAM
- With argument: shows RAM for specific node only
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
SNAP sync is broken for Linea - it picks an old pivot block (~24.7M) that
no peers can serve world state for. This causes:
- World state download stuck with 0 pending requests
- Maru unable to push blocks to EL without complete world state
- Node stuck returning block 0 for "latest"
FULL sync executes every block from genesis. It's slower but reliable
and allows maru to drive the sync via engine API.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When Besu is behind the CL head (e.g., during initial sync or after restart),
desync-tolerance=0 prevents maru from sending any fork choice updates to Besu.
This causes Besu to remain stuck at its current block.
Increasing desync-tolerance to 100000 allows maru to continue sending blocks
even when Besu is significantly behind, enabling it to catch up.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
robandClaude Fable 5.1