dshackle's SIGHUP reload silently fails for upstream REMOVALS (and method-set
changes, same remove+add path): it logs "Config is not reloaded, cause - ...",
updates the in-memory config but leaves runtime selectors stale (drift), and keeps
serving the old config. No runtime admin API exists to mutate upstreams, so a
container restart is the only way to actually apply such changes.
Replace the blind SIGHUP with: per-container SIGHUP, then check post-SIGHUP logs
for the failure marker ("Config is not reloaded, cause -") or a dropped concurrent
HUP ("Reloading is in progress"); on either, docker restart to load config fresh.
"Reloading config has been completed" is NOT success (finally-block) — deliberately
not treated as such. Exit 0 when config ends up applied (via SIGHUP or restart);
non-zero only if a needed restart fails or the container doesn't come back.
Layered on top of the override-weave (apply-dshackle-overrides.py) and the 1:1
invariant gate (validate-dshackle-1to1.py) — both run unchanged before the reload.
Co-Authored-By: Claude <noreply@anthropic.com>
The splits completeness floor is TIME-based (trust hours complete since shipper
connect) and blind to event-loss-by-rotation (2026-07-10: raw_events rotated
every 1-2min, cruncher lost ~85% of us-32 events, floor still called the hours
complete). Shipper now injects {"__shipstat__":1,"sess":..,"sent":N,"ts":..}
every 1000 lines or 30s through the SAME tcp pipe. Cruncher diffs shipper-sent
vs DB-ingested per gateway/hour -> a reconciliation-based floor that catches the
loss a clock cannot. sess=connection-start epoch so resets on reconnect are
detectable. Logic in a mounted .awk file (not inline) to dodge sh/awk quoting.
Verified on busybox:1.36 (systime/fflush OK). NOT deployed until claude@'s
cruncher filters __shipstat__ from event parsing - coordinating via SIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same duplicated listener code, same two latent bugs. Not exercised today
(de-32 target has SLOWDISK=False) but any clone to a SLOWDISK=True host
would have failed identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The slowdisk --transform args contain '|' and quotes; the ssh+screen+bash -c
stack strips one quoting level too many and splits the pipeline (broke the
katana retry within minutes of fccf2792). A heredoc-generated
/tmp/transfer_<key>.sh is parsed exactly once on the target. Applies to both
the screen and nohup listener branches; the direct-ssh fallback already had a
single shell layer and stays inline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tar cannot create files through a directory symlink whose target is on
another device: open() fails with 'Invalid cross-device link' even under
--keep-directory-symlink (reproduced, GNU tar 1.34 and 1.35). This killed
the katana->uk-4 provision (place-provision-katana-EU-NorthWest-p1) and
means the slowdisk EXTRACT path never worked on hosts with a separate
/slowdisk device. Rewrite member paths with --transform so tar writes
directly into /slowdisk/rpc_<key>__data_<rel>; the symlink stays for the
container's runtime view. Also aligns restore-volumes' nested-path target
naming to the double-underscore convention clone-backup already uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified from crypto-org-chain/cronos-testnets: genesis, 3 official seeds,
t3 statesync RPC, basetcro denom; official daily snapshots exist for restore
(sidesteps the cosmos binary-upgrade gauntlet). Closes the planner's
'cronos-testnet: no known node_path' escalation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/root/rpc-local/dshackle-overrides.yaml (host-local, like rpc-local/compose.d) is
re-woven into main_configs/*.yaml by reload_dshackle.sh on every config push.
Method-disable decisions are local to each gateway's market (operator eviction
rule: cost_ratio = time_share/call_share at flat pay); this makes them durable
inputs instead of fragile output patches. Step 1 of moving dshackle config
generation onto the gateway hosts (step 2: ssh forced-command upstreams fetch,
generic dshackle.yaml include list derived from drpc-gateways.json).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each gateway serves its own regional market, so the eviction economics
(cost_ratio = time_share/call_share at flat per-call pay) are computed where
they apply: against the local dshackle's histograms. Silent on non-gateway
hosts. Offenders = candidates for per-upstream method-groups/methods disables
(operator eviction rule 2026-07-10). sendRawTransaction et al [protected].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All defaults preserve current effective behavior - this deploy is a rendering no-op
until a host_var overrides a value. Part of the resource-optimizer knob rollout
(resource-optimizer/knob-manifest.yaml is the registry; verify_manifest.py enforces
default sync between templates and envfile).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dshackle 0.79.10's beacon health validator (GET /eth/v1/node/health via
GenericSingleCallValidator) times out after 5000ms on the spec-compliant
empty-body 200 that nimbus returns, marking the upstream UNAVAILABLE in a
flap loop; live requests routed during flap windows fail 'Source was empty'.
This is the root cause behind dRPC's 617k failed validators POSTs complaint
(2026-07-09). Reproduced + workaround verified on an isolated dshackle
instance on de-14. Head tracking still governs availability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decoupled by design: dshackle writes the file regardless; a dead listener
or slow link stalls only the busybox sidecar. Retry resumes at file end
(bounded loss); the listener-side prometheus reconciliation surfaces gaps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gateway request-splits pilot (earnings misattribution fix): the accessLog
stub in main_configs pointed at /logs which was never mounted. Host-side
dir lets the splits tailer aggregate per-caller response counts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A silent fallback to primary-disk extraction violates the caller's NVMe
capacity math now that the planner treats slowdisk as a separate pool.
When SLOWDISK=True and the manifest's static sizes (accurate, replacing
the whole-archive x2 estimate that false-refuses large restores) do not
fit above the floor, error out with the explicit remedy: re-run with
--no-slowdisk as the conscious override. Operator directive 2026-07-09.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live test on rpc-uk-4 (unichain-mainnet pruned, 166G static_files moved to
/slowdisk + symlinked): reth healed consistency, initialized the
StaticFileProducer, and resumed committing canonical blocks - full
read/write through the symlink. The guard's premise (old reth refusing
symlinked static_files) no longer holds; pre-v2 reths fail loudly at start,
fallback is --no-slowdisk. Restores/clones of reth nodes to SLOWDISK=True
hosts now offload statics automatically (NVMe is the scarce resource).
Note: reth also has --datadir.static-files for a flag-based layout later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Candidate fix for the receipts RLP decode failure that blocks snap sync
past block 4,178,637 (2026-04-12): peers dropped with 'invalid message
(code 10) receipt 1: rlp: ListEnd not positioned at EOL' on the previous
newest published image.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fenced nodes are dropped from COMPOSE_FILE so show-status silently omitted
them - a fenced-but-running node looked 'gone'. rpc-update now writes
/root/rpc/.fenced (node | until | reason | owner per active window); print
it as a footer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 1MB near-empty guard (added for purged-volume zstd stubs) rejected every
legitimate _config volume archive (cronos: 68KB), so any node whose registry
entry includes a _config volume reported Restorable: 0 despite healthy
multi-GB data backups (cronos had three; 14 registry entries carry _config
volumes). _config archives now pass at >=4KB; data volumes keep the 1MB floor.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hemi cannot full-sync across old BTC reorg points: EVM blocks referencing
orphaned Bitcoin Attributes ancestors are unexecutable once no peer retains
the stale BTC block (us-16 wedged at 3353463 on reorged-out block
00000000000000000001dcb3...b971). hemi's official l2-config.toml ships
SyncMode=snap. Registry: 150 entries gain node names (regen enrichment).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Faithful copy of VIBE.md to seed cursor-specific tuning. The implement-vibe-node-change
wrapper, when run with VN_ENGINE=cursor (now the default), points cursor-agent at this
doc instead of VIBE.md so the two backends' instructions can diverge over time.
heimdall crash-looped on 'unknown flag: --rest-server.addr'. Per heimdalld start
--help, the bone-fork binary takes --rest-server (bool) to enable REST/LCD and
--laddr (default tcp://0.0.0.0:1317) for its listen address — there is no
--rest-server.addr. Swap it. (The rest_server-injection + idempotent-init fixes
from the prior PR worked: init now skips re-init, fetches the heimdall-109 genesis,
no more 'rest_server already defined'.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
heimdall crash-looped on 'toml: key rest_server is already defined'. init.sh
inserted 'rest_server = true' after [rpc] on EVERY boot (non-idempotent), piling up
duplicates; rest_server is not a valid CometBFT config.toml key anyway — REST is
enabled by the --rest-server flag in the start command.
- Remove the rest_server / rest_server_addr config injection.
- Self-heal: strip any previously-injected rest_server lines (so already-corrupted
volumes recover on next boot — no volume wipe needed).
- Guard heimdalld init to first boot only (config.toml absent) so restarts don't
reset node identity or re-parse the existing config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A parallel change reverted this to FROM shibaone/heimdall:v1.0.7-bone — an image
that exists on no registry, so the heimdall build fails. Restore the source build:
clone shibaone/heimdall@${CL_VERSION}, make install, then layer the CometBFT init
entrypoint. Pairs with vibe-node PR (node_build + standalone service name).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
shibaone ships no docker images for the bone fork — only source + .deb config
packages — so both Dockerfiles must clone+build, not FROM a (nonexistent) image.
- bor.Dockerfile: clone shibaone/bor@${BOR_VERSION}, make bor, cp build/bin/bor
(was: alpine + wrong /src/build/bor path). golang:1.22.1 like upstream.
- cometbft.Dockerfile: clone shibaone/heimdall@${CL_VERSION}, make install
(was: FROM shibaone/heimdall:v1.0.7-bone — that tag does not exist on any
registry), then layer the CometBFT init entrypoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A hung check-health.sh (aztec-testnet, looping on an unresponsive reference RPC)
blocked show-status.sh's parallel 'wait' for 3.5h, hanging the whole fleet
rpc-update and holding the deploy lock. Each curl was bounded (-m 3) and the
retry loop capped (3x), but the call itself wasn't time-bounded.
- sync-status.sh: wrap each check-health.sh call in 'timeout ${HC_TIMEOUT:-30}'
(-> exit 124 + 'timeout' status on overrun).
- show-status.sh: wrap the whole per-node sync-status.sh call in
'timeout ${SYNC_TIMEOUT:-60}' so the parallel wait can never block forever.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- BUG 1: Add error checking after tar extraction for both LOCAL and REMOTE-CACHE branches
- Check exit status of tar -I zstd -xf commands
- Print error to stderr and exit non-zero on failure
- Prevents silent truncation where corrupt/incomplete backup extracts partial data
- Mirrors existing remote-STREAM branch error handling
- BUG 2: Fix REMOTE branch to resume incomplete aria2c downloads
- Check for presence of <file>.aria2 control file as incomplete signal
- aria2c -c continues/resumes download when .aria2 file exists
- Only skip download when file exists AND no .aria2 control file remains
- aria2 deletes .aria2 sidecar on successful completion, making it a reliable signal
- Maintain all existing flags: aria2c -c -Z -x8 -j8 -s8 -d
- Preserve reth guard logic and static-file offload behavior unchanged
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add RETH GUARD to clone-backup.sh and clone-node.sh: when the config name
contains 'reth', skip the whole /slowdisk static-file symlink offload and
extract everything onto the primary disk (equivalent to --no-slowdisk).
This matches the already-correct restore-volumes.sh behavior.
Reason: reth refuses to start when its static_files directory is a symlink,
failing at boot with 'failed to create dir static_files: File exists'.
- Add --keep-directory-symlink to all tar extraction options in both scripts
for the SLOWDISK path. This allows tar to extract files THROUGH the
pre-created directory symlinks instead of trying to mkdir over them
(which fails with 'Cannot mkdir: File exists'). This matches the
already-correct restore-volumes.sh behavior.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Rename doma testnet relay from celestia to op-alt (ghcr.io/celestiaorg/op-alt-da)
- Add op-alt relay for doma mainnet with image tag 0.15.0 (not v0.15.0)
- Vendor complete mainnet rollup.json with alt_da (GenericCommitment)
- Fix per-network relay settings: namespace + CELESTIA_*_RPC env vars
- Switch mainnet op-geth to use vendored rollup.json (bind-mounted)
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
- Add relay template (templates/relays/celestia.doma.yml) for op-alt-da
- Add celestia relay config to op/doma stack in context.yml
- Add relay: celestia to op/doma testnet in config.yml
- Add op-node.doma.yml template with ALTDA env vars gated on relay_name
- Update op/doma/testnet/rollup.json with complete config including alt_da block
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Revert to the prune-cycle model for minimal nodes: the normal compose serves
RPC with no --init.prune, and a generated .prune.yml runs --init.prune=minimal,
driven periodically by prune-if-prunable (same mechanism as pruned/full). Minimal
nodes are seeded from a pruned backup, then pruned to minimal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct the minimal-node model: the minimal compose itself carries
--init.prune=minimal (prunes to genesis+head on start) instead of relying on a
separate .prune.yml — otherwise a minimal node was byte-identical to pruned in
normal operation and had no way to enforce minimal state. Removed the redundant
minimal .prune.yml files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New minimal profile for arbitrum one/nova/sepolia: a pruned-style node whose
prune cycle uses --init.prune=minimal (most aggressive: genesis+head only)
instead of full. Separate composes so it can be tested independently of the
production pruned nodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes prune-version drift that corrupted nodes (prune ran an older nitro binary
than the node). All .prune.yml now regenerate from the same config as the normal
compose via client_needs_prune.
- arb nitro: nova/one/sepolia .prune.yml bumped to the node's version (v3.10.1)
- avalanche: add .prune.yml variants that mount a /config/prune chain-config with
offline-pruning-enabled, plus avalanche/{mainnet,fuji}/prune/C/config.json
- archive profiles are excluded (db_type==pruned gate) so they are never pruned
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the restore-volumes.sh --no-slowdisk capability for live/backup clones.
Both scripts gate the target /slowdisk static-file offload on the target's
SLOWDISK env (case-insensitive, matches the Python-templated 'True') and accept
a --no-slowdisk flag that forces the offload off for one run. When SLOWDISK is
on but the target /slowdisk is too small for the static files, the clone warns
and aborts, telling the operator to re-run with --no-slowdisk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Use COMPOSE_FILE from .env to get list of nodes
- Match containers by service labels from compose files
- Output format now matches show-status (node paths, not container IDs)
- Sorted by RAM usage descending
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Path maps directly to compose file in subdirectories:
op/reth/base-mainnet-op-reth-archive-trace -> op/reth/base-mainnet-op-reth-archive-trace.yml
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix: handle full path like ethereum/geth/node-name (extract basename)
- Without args: show RAM per node (grouped by compose project), not per container
- Sorted by RAM usage descending
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Without argument: shows all containers sorted by RAM, total container RAM, server RAM
- With argument: shows RAM for specific node only
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
SNAP sync is broken for Linea - it picks an old pivot block (~24.7M) that
no peers can serve world state for. This causes:
- World state download stuck with 0 pending requests
- Maru unable to push blocks to EL without complete world state
- Node stuck returning block 0 for "latest"
FULL sync executes every block from genesis. It's slower but reliable
and allows maru to drive the sync via engine API.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When Besu is behind the CL head (e.g., during initial sync or after restart),
desync-tolerance=0 prevents maru from sending any fork choice updates to Besu.
This causes Besu to remain stuck at its current block.
Increasing desync-tolerance to 100000 allows maru to continue sending blocks
even when Besu is significantly behind, enabling it to catch up.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When payload-validation-enabled is true, maru validates every block against
Besu before sending fork choice updates. If Besu is in an inconsistent state
(e.g., stuck in SNAP sync), this causes maru to stop sending fork choice
updates entirely, preventing Besu from ever syncing.
The official Linea configuration uses payload-validation-enabled = false,
which allows maru to continue sending fork choice updates regardless of
Besu's current state.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Set payload-validation-enabled=true in Maru config to ensure
payloads are sent to the execution client. Without this, Maru
doesn't send forkchoice/newPayload calls when EL reports synced
status (even at block 0).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add linea/geth/ compose files (mainnet/sepolia, pruned/archive)
- Update Maru version and --network flag in besu/erigon3 configs
- Update compose_registry.json
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Upgrade from 9737a45 to v1.0.0-20260108114606-36f5e2f
- Use --network=linea-mainnet for built-in config
- May fix advertise-ip issue for peer discovery
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Juno returns chain ID as hex-encoded ASCII (0x534e5f5345504f4c4941)
rather than plain string (SN_SEPOLIA). Match both formats.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Detect Starknet paths and use starknet_getBlockWithTxHashes
- Return decimal block_number directly instead of hex conversion
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- sync-status.sh now detects Starknet paths and uses starknet_chainId
- Maps SN_MAIN/SN_SEPOLIA chain IDs to reference endpoints
- check-health.sh accepts --starknet flag for Starknet mode
- Uses starknet_getBlockWithTxHashes instead of eth_getBlockByNumber
- Handles decimal timestamps and block_hash field differences
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Service name simplified to rootstock-mainnet (no -client suffix)
- Traefik middlewares handle Host:localhost header rewriting
- Proper WS routing on port 8546
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace nginx proxy with traefik headers middleware for Host rewriting.
Fixes container IP mismatch issues on container restart.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
robandClaude