dshackle's SIGHUP reload silently fails for upstream REMOVALS (and method-set
changes, same remove+add path): it logs "Config is not reloaded, cause - ...",
updates the in-memory config but leaves runtime selectors stale (drift), and keeps
serving the old config. No runtime admin API exists to mutate upstreams, so a
container restart is the only way to actually apply such changes.
Replace the blind SIGHUP with: per-container SIGHUP, then check post-SIGHUP logs
for the failure marker ("Config is not reloaded, cause -") or a dropped concurrent
HUP ("Reloading is in progress"); on either, docker restart to load config fresh.
"Reloading config has been completed" is NOT success (finally-block) — deliberately
not treated as such. Exit 0 when config ends up applied (via SIGHUP or restart);
non-zero only if a needed restart fails or the container doesn't come back.
Layered on top of the override-weave (apply-dshackle-overrides.py) and the 1:1
invariant gate (validate-dshackle-1to1.py) — both run unchanged before the reload.
Co-Authored-By: Claude <noreply@anthropic.com>
Insert validate-dshackle-1to1.py between the overrides weave and the HUP signal:
a dshackle config routing >1 node for the same chain is REFUSED (exit 1) so the
live config stays active. Structural enforcement of 1 dshackle = 1 node/chain —
we can't attribute traffic to multiple nodes behind one proxy (no per-upstream
request metric; conn-seconds biases it). See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Refuses to activate a dshackle config routing >1 upstream for the same chain
(any kind/client) — we cannot attribute traffic to multiple nodes behind one
proxy (no per-upstream request-count metric; conn-seconds biases it), and the
attribution model + placement planner both assume 1:1. Wire into
reload_dshackle.sh BEFORE the reload so a drifting config can never go live.
Host-local by design (the repo doesn't track host->dshackle assignment; 1:1 is
only checkable against the assembled dshackle config at /root/rpc/main_configs).
See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Empty ws-origins rejects WebSocket connections with an Origin header
(dshackle head-subscription monitoring). Sed under [json-rpc] like
address/ws-address so restarts keep the value.
The splits completeness floor is TIME-based (trust hours complete since shipper
connect) and blind to event-loss-by-rotation (2026-07-10: raw_events rotated
every 1-2min, cruncher lost ~85% of us-32 events, floor still called the hours
complete). Shipper now injects {"__shipstat__":1,"sess":..,"sent":N,"ts":..}
every 1000 lines or 30s through the SAME tcp pipe. Cruncher diffs shipper-sent
vs DB-ingested per gateway/hour -> a reconciliation-based floor that catches the
loss a clock cannot. sess=connection-start epoch so resets on reconnect are
detectable. Logic in a mounted .awk file (not inline) to dodge sh/awk quoting.
Verified on busybox:1.36 (systime/fflush OK). NOT deployed until claude@'s
cruncher filters __shipstat__ from event parsing - coordinating via SIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same duplicated listener code, same two latent bugs. Not exercised today
(de-32 target has SLOWDISK=False) but any clone to a SLOWDISK=True host
would have failed identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The slowdisk --transform args contain '|' and quotes; the ssh+screen+bash -c
stack strips one quoting level too many and splits the pipeline (broke the
katana retry within minutes of fccf2792). A heredoc-generated
/tmp/transfer_<key>.sh is parsed exactly once on the target. Applies to both
the screen and nohup listener branches; the direct-ssh fallback already had a
single shell layer and stays inline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tar cannot create files through a directory symlink whose target is on
another device: open() fails with 'Invalid cross-device link' even under
--keep-directory-symlink (reproduced, GNU tar 1.34 and 1.35). This killed
the katana->uk-4 provision (place-provision-katana-EU-NorthWest-p1) and
means the slowdisk EXTRACT path never worked on hosts with a separate
/slowdisk device. Rewrite member paths with --transform so tar writes
directly into /slowdisk/rpc_<key>__data_<rel>; the symlink stays for the
container's runtime view. Also aligns restore-volumes' nested-path target
naming to the double-underscore convention clone-backup already uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node was forked at block 30194654 because rollup.json lacked holocene_time
and later fork timestamps; add granite/holocene/isthmus/jovian times so
op-node can apply post-Holocene batches after resync.
Verified from crypto-org-chain/cronos-testnets: genesis, 3 official seeds,
t3 statesync RPC, basetcro denom; official daily snapshots exist for restore
(sidesteps the cosmos binary-upgrade gauntlet). Closes the planner's
'cronos-testnet: no known node_path' escalation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/root/rpc-local/dshackle-overrides.yaml (host-local, like rpc-local/compose.d) is
re-woven into main_configs/*.yaml by reload_dshackle.sh on every config push.
Method-disable decisions are local to each gateway's market (operator eviction
rule: cost_ratio = time_share/call_share at flat pay); this makes them durable
inputs instead of fragile output patches. Step 1 of moving dshackle config
generation onto the gateway hosts (step 2: ssh forced-command upstreams fetch,
generic dshackle.yaml include list derived from drpc-gateways.json).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each gateway serves its own regional market, so the eviction economics
(cost_ratio = time_share/call_share at flat per-call pay) are computed where
they apply: against the local dshackle's histograms. Silent on non-gateway
hosts. Offenders = candidates for per-upstream method-groups/methods disables
(operator eviction rule 2026-07-10). sendRawTransaction et al [protected].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All defaults preserve current effective behavior - this deploy is a rendering no-op
until a host_var overrides a value. Part of the resource-optimizer knob rollout
(resource-optimizer/knob-manifest.yaml is the registry; verify_manifest.py enforces
default sync between templates and envfile).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dshackle 0.79.10's beacon health validator (GET /eth/v1/node/health via
GenericSingleCallValidator) times out after 5000ms on the spec-compliant
empty-body 200 that nimbus returns, marking the upstream UNAVAILABLE in a
flap loop; live requests routed during flap windows fail 'Source was empty'.
This is the root cause behind dRPC's 617k failed validators POSTs complaint
(2026-07-09). Reproduced + workaround verified on an isolated dshackle
instance on de-14. Head tracking still governs availability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
