EL/CL baseFee mismatch at block 22961752; CL crash-loops on
INVALID payload from post-v1.4.2 block format while beacon-kit
is on v1.4.1. Also regenerates ink-mainnet OPE P2P bootnodes
from parent a790c3e and refreshes reference-rpc-endpoint (mova).
Co-authored-by: Cursor <cursoragent@cursor.com>
-jwt volumes (EL/CL shared secret) are regenerated at create-node and never
backed up, same as -secrets. Two fixes:
- clone-backup.sh: extend the fd8dd15b secrets skip to *-jwt (both paths)
- list-restorable.sh: exclude *-jwt/*-secrets from the all-volumes-required
restorability check. Both plasma composes showed Restorable: 0 for days
(mainnet 407G + testnet 98G+29G data backups present since 07-18/19)
purely because rpc_plasma-*-jwt had no backup; this misled the 07-16
plasma restore assessment ('de-32: NO plasma backup').
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Secrets volumes are regenerated at create-node and generally have no
backup by design. Counting the skip as Failed (exit 1) false-failed
whole provisions whose data volumes transferred fine (tempo de-32->de-27,
job e9634e66: executor's broken-pipe guard reset the action and would
have re-cloned everything). Reported by claude@ (inbox 829bc624).
- no-backup on a *-secrets volume -> return 2 (skip), both nc and ssh paths
- rc=2 tracked as Skipped in the summary, excluded from the failure count
- exit status now keys on failed_volumes only
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The false default forced every op-node restart to walk back to the finalized head
before driving the EL. On nodes whose derivation lags days behind their serving
gossip head (hashkey us-40: 510k blocks / 12 days), that froze the EL for hours
while dRPC de-routed the sole US provider. true resumes at the unsafe head in
minutes; safe/finalized still verify and catch up in the background. ronin
already ran true per-chain (same lesson); xlayer keeps its deliberate false pin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the alpine genesis/rollup init containers and their
service_completed_successfully gate for all hashkeychain variants. The gate
wedged us-40: a reconcile compose-up recreated the op-reth EL but not the
init-gated op-node CL -> fresh EL + stale CL -> 2h engine freeze -> sole-provider
market -> $0 (2026-07-21). genesis.json now committed beside the existing
rollup.json so render_templates auto-detects custom_genesis/custom_rollup and
binds the local files; CL now depends only on the EL (service_started,
restart:true), identical to op-mainnet/op-sepolia. No S3 on the boot path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Image default is -Xms1024M with no -Xmx, so heap can grow unbounded (~30GiB).
us-41 was at 90% memory with no swap; -Xmx4g provides a safety limit.
Co-authored-by: Cursor <cursoragent@cursor.com>
Catch up from v1.2.1/1.0.1 pins so mainnet and sepolia followers track the current Consensys/maru release.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add bootstrap-peers.sh and fall back to stakesquid.eu/bootstrap when local peerset capture is empty; optional --seed-bootstrap on connect-peers.
Co-authored-by: Cursor <cursoragent@cursor.com>
Re-label consensys/maru FROM tag into image OCI labels (upstream leaves
version/vcs-ref empty) and scrape /metrics on :9090 instead of the 404ing
/debug/metrics/prometheus path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Canonicalize plasma-testnet so the tracked compose adopts the same
volume/service names as the de-32 hand-written override (rpc_plasma-
testnet-reth-pruned-trace-{execution,consensus,jwt}). Also refresh
mainnet observer sed to be chain-agnostic.
Hosts like rpc-us-32 put /var/lib/docker (or volumes/) on another mount;
tar -C / then EXDEV on every var/lib/docker/volumes/... member. Canonicalize
the deepest static prefix (readlink -f /var/lib/docker/volumes) and append a
--transform after SLOWDISK_TRANSFORMS. Keying only on /var/lib/docker misses
a real docker root with volumes/ as the symlink. No-op on normal hosts.
Per-volume _data static-file offload stays on SLOWDISK_TRANSFORMS.
Co-authored-by: Cursor <cursoragent@cursor.com>
Also regenerate compose for already-merged hashkey static-peers / op-geth
static-nodes entrypoint (parent #1437) that had not yet landed in rpc.
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace buggy && short-circuit with if/fi so the one-shot init
exits 0 when the signing key is already present (matches mainnet/
moderato and templates/clients/reth.tempo.yml), unblocking sync
resume on data-intact hosts still on the orphaned testnet compose.
dshackle's SIGHUP reload silently fails for upstream REMOVALS (and method-set
changes, same remove+add path): it logs "Config is not reloaded, cause - ...",
updates the in-memory config but leaves runtime selectors stale (drift), and keeps
serving the old config. No runtime admin API exists to mutate upstreams, so a
container restart is the only way to actually apply such changes.
Replace the blind SIGHUP with: per-container SIGHUP, then check post-SIGHUP logs
for the failure marker ("Config is not reloaded, cause -") or a dropped concurrent
HUP ("Reloading is in progress"); on either, docker restart to load config fresh.
"Reloading config has been completed" is NOT success (finally-block) — deliberately
not treated as such. Exit 0 when config ends up applied (via SIGHUP or restart);
non-zero only if a needed restart fails or the container doesn't come back.
Layered on top of the override-weave (apply-dshackle-overrides.py) and the 1:1
invariant gate (validate-dshackle-1to1.py) — both run unchanged before the reload.
Co-Authored-By: Claude <noreply@anthropic.com>
Insert validate-dshackle-1to1.py between the overrides weave and the HUP signal:
a dshackle config routing >1 node for the same chain is REFUSED (exit 1) so the
live config stays active. Structural enforcement of 1 dshackle = 1 node/chain —
we can't attribute traffic to multiple nodes behind one proxy (no per-upstream
request metric; conn-seconds biases it). See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Refuses to activate a dshackle config routing >1 upstream for the same chain
(any kind/client) — we cannot attribute traffic to multiple nodes behind one
proxy (no per-upstream request-count metric; conn-seconds biases it), and the
attribution model + placement planner both assume 1:1. Wire into
reload_dshackle.sh BEFORE the reload so a drifting config can never go live.
Host-local by design (the repo doesn't track host->dshackle assignment; 1:1 is
only checkable against the assembled dshackle config at /root/rpc/main_configs).
See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
