Canonicalize plasma-testnet so the tracked compose adopts the same
volume/service names as the de-32 hand-written override (rpc_plasma-
testnet-reth-pruned-trace-{execution,consensus,jwt}). Also refresh
mainnet observer sed to be chain-agnostic.
Hosts like rpc-us-32 put /var/lib/docker (or volumes/) on another mount;
tar -C / then EXDEV on every var/lib/docker/volumes/... member. Canonicalize
the deepest static prefix (readlink -f /var/lib/docker/volumes) and append a
--transform after SLOWDISK_TRANSFORMS. Keying only on /var/lib/docker misses
a real docker root with volumes/ as the symlink. No-op on normal hosts.
Per-volume _data static-file offload stays on SLOWDISK_TRANSFORMS.
Co-authored-by: Cursor <cursoragent@cursor.com>
Also regenerate compose for already-merged hashkey static-peers / op-geth
static-nodes entrypoint (parent #1437) that had not yet landed in rpc.
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace buggy && short-circuit with if/fi so the one-shot init
exits 0 when the signing key is already present (matches mainnet/
moderato and templates/clients/reth.tempo.yml), unblocking sync
resume on data-intact hosts still on the orphaned testnet compose.
dshackle's SIGHUP reload silently fails for upstream REMOVALS (and method-set
changes, same remove+add path): it logs "Config is not reloaded, cause - ...",
updates the in-memory config but leaves runtime selectors stale (drift), and keeps
serving the old config. No runtime admin API exists to mutate upstreams, so a
container restart is the only way to actually apply such changes.
Replace the blind SIGHUP with: per-container SIGHUP, then check post-SIGHUP logs
for the failure marker ("Config is not reloaded, cause -") or a dropped concurrent
HUP ("Reloading is in progress"); on either, docker restart to load config fresh.
"Reloading config has been completed" is NOT success (finally-block) — deliberately
not treated as such. Exit 0 when config ends up applied (via SIGHUP or restart);
non-zero only if a needed restart fails or the container doesn't come back.
Layered on top of the override-weave (apply-dshackle-overrides.py) and the 1:1
invariant gate (validate-dshackle-1to1.py) — both run unchanged before the reload.
Co-Authored-By: Claude <noreply@anthropic.com>
Insert validate-dshackle-1to1.py between the overrides weave and the HUP signal:
a dshackle config routing >1 node for the same chain is REFUSED (exit 1) so the
live config stays active. Structural enforcement of 1 dshackle = 1 node/chain —
we can't attribute traffic to multiple nodes behind one proxy (no per-upstream
request metric; conn-seconds biases it). See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Refuses to activate a dshackle config routing >1 upstream for the same chain
(any kind/client) — we cannot attribute traffic to multiple nodes behind one
proxy (no per-upstream request-count metric; conn-seconds biases it), and the
attribution model + placement planner both assume 1:1. Wire into
reload_dshackle.sh BEFORE the reload so a drifting config can never go live.
Host-local by design (the repo doesn't track host->dshackle assignment; 1:1 is
only checkable against the assembled dshackle config at /root/rpc/main_configs).
See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Empty ws-origins rejects WebSocket connections with an Origin header
(dshackle head-subscription monitoring). Sed under [json-rpc] like
address/ws-address so restarts keep the value.
The splits completeness floor is TIME-based (trust hours complete since shipper
connect) and blind to event-loss-by-rotation (2026-07-10: raw_events rotated
every 1-2min, cruncher lost ~85% of us-32 events, floor still called the hours
complete). Shipper now injects {"__shipstat__":1,"sess":..,"sent":N,"ts":..}
every 1000 lines or 30s through the SAME tcp pipe. Cruncher diffs shipper-sent
vs DB-ingested per gateway/hour -> a reconciliation-based floor that catches the
loss a clock cannot. sess=connection-start epoch so resets on reconnect are
detectable. Logic in a mounted .awk file (not inline) to dodge sh/awk quoting.
Verified on busybox:1.36 (systime/fflush OK). NOT deployed until claude@'s
cruncher filters __shipstat__ from event parsing - coordinating via SIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same duplicated listener code, same two latent bugs. Not exercised today
(de-32 target has SLOWDISK=False) but any clone to a SLOWDISK=True host
would have failed identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The slowdisk --transform args contain '|' and quotes; the ssh+screen+bash -c
stack strips one quoting level too many and splits the pipeline (broke the
katana retry within minutes of fccf2792). A heredoc-generated
/tmp/transfer_<key>.sh is parsed exactly once on the target. Applies to both
the screen and nohup listener branches; the direct-ssh fallback already had a
single shell layer and stays inline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tar cannot create files through a directory symlink whose target is on
another device: open() fails with 'Invalid cross-device link' even under
--keep-directory-symlink (reproduced, GNU tar 1.34 and 1.35). This killed
the katana->uk-4 provision (place-provision-katana-EU-NorthWest-p1) and
means the slowdisk EXTRACT path never worked on hosts with a separate
/slowdisk device. Rewrite member paths with --transform so tar writes
directly into /slowdisk/rpc_<key>__data_<rel>; the symlink stays for the
container's runtime view. Also aligns restore-volumes' nested-path target
naming to the double-underscore convention clone-backup already uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node was forked at block 30194654 because rollup.json lacked holocene_time
and later fork timestamps; add granite/holocene/isthmus/jovian times so
op-node can apply post-Holocene batches after resync.
