The splits completeness floor is TIME-based (trust hours complete since shipper
connect) and blind to event-loss-by-rotation (2026-07-10: raw_events rotated
every 1-2min, cruncher lost ~85% of us-32 events, floor still called the hours
complete). Shipper now injects {"__shipstat__":1,"sess":..,"sent":N,"ts":..}
every 1000 lines or 30s through the SAME tcp pipe. Cruncher diffs shipper-sent
vs DB-ingested per gateway/hour -> a reconciliation-based floor that catches the
loss a clock cannot. sess=connection-start epoch so resets on reconnect are
detectable. Logic in a mounted .awk file (not inline) to dodge sh/awk quoting.
Verified on busybox:1.36 (systime/fflush OK). NOT deployed until claude@'s
cruncher filters __shipstat__ from event parsing - coordinating via SIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same duplicated listener code, same two latent bugs. Not exercised today
(de-32 target has SLOWDISK=False) but any clone to a SLOWDISK=True host
would have failed identically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The slowdisk --transform args contain '|' and quotes; the ssh+screen+bash -c
stack strips one quoting level too many and splits the pipeline (broke the
katana retry within minutes of fccf2792). A heredoc-generated
/tmp/transfer_<key>.sh is parsed exactly once on the target. Applies to both
the screen and nohup listener branches; the direct-ssh fallback already had a
single shell layer and stays inline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tar cannot create files through a directory symlink whose target is on
another device: open() fails with 'Invalid cross-device link' even under
--keep-directory-symlink (reproduced, GNU tar 1.34 and 1.35). This killed
the katana->uk-4 provision (place-provision-katana-EU-NorthWest-p1) and
means the slowdisk EXTRACT path never worked on hosts with a separate
/slowdisk device. Rewrite member paths with --transform so tar writes
directly into /slowdisk/rpc_<key>__data_<rel>; the symlink stays for the
container's runtime view. Also aligns restore-volumes' nested-path target
naming to the double-underscore convention clone-backup already uses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node was forked at block 30194654 because rollup.json lacked holocene_time
and later fork timestamps; add granite/holocene/isthmus/jovian times so
op-node can apply post-Holocene batches after resync.
Verified from crypto-org-chain/cronos-testnets: genesis, 3 official seeds,
t3 statesync RPC, basetcro denom; official daily snapshots exist for restore
(sidesteps the cosmos binary-upgrade gauntlet). Closes the planner's
'cronos-testnet: no known node_path' escalation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/root/rpc-local/dshackle-overrides.yaml (host-local, like rpc-local/compose.d) is
re-woven into main_configs/*.yaml by reload_dshackle.sh on every config push.
Method-disable decisions are local to each gateway's market (operator eviction
rule: cost_ratio = time_share/call_share at flat pay); this makes them durable
inputs instead of fragile output patches. Step 1 of moving dshackle config
generation onto the gateway hosts (step 2: ssh forced-command upstreams fetch,
generic dshackle.yaml include list derived from drpc-gateways.json).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each gateway serves its own regional market, so the eviction economics
(cost_ratio = time_share/call_share at flat per-call pay) are computed where
they apply: against the local dshackle's histograms. Silent on non-gateway
hosts. Offenders = candidates for per-upstream method-groups/methods disables
(operator eviction rule 2026-07-10). sendRawTransaction et al [protected].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All defaults preserve current effective behavior - this deploy is a rendering no-op
until a host_var overrides a value. Part of the resource-optimizer knob rollout
(resource-optimizer/knob-manifest.yaml is the registry; verify_manifest.py enforces
default sync between templates and envfile).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dshackle 0.79.10's beacon health validator (GET /eth/v1/node/health via
GenericSingleCallValidator) times out after 5000ms on the spec-compliant
empty-body 200 that nimbus returns, marking the upstream UNAVAILABLE in a
flap loop; live requests routed during flap windows fail 'Source was empty'.
This is the root cause behind dRPC's 617k failed validators POSTs complaint
(2026-07-09). Reproduced + workaround verified on an isolated dshackle
instance on de-14. Head tracking still governs availability.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Decoupled by design: dshackle writes the file regardless; a dead listener
or slow link stalls only the busybox sidecar. Retry resumes at file end
(bounded loss); the listener-side prometheus reconciliation surfaces gaps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gateway request-splits pilot (earnings misattribution fix): the accessLog
stub in main_configs pointed at /logs which was never mounted. Host-side
dir lets the splits tailer aggregate per-caller response counts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A silent fallback to primary-disk extraction violates the caller's NVMe
capacity math now that the planner treats slowdisk as a separate pool.
When SLOWDISK=True and the manifest's static sizes (accurate, replacing
the whole-archive x2 estimate that false-refuses large restores) do not
fit above the floor, error out with the explicit remedy: re-run with
--no-slowdisk as the conscious override. Operator directive 2026-07-09.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live test on rpc-uk-4 (unichain-mainnet pruned, 166G static_files moved to
/slowdisk + symlinked): reth healed consistency, initialized the
StaticFileProducer, and resumed committing canonical blocks - full
read/write through the symlink. The guard's premise (old reth refusing
symlinked static_files) no longer holds; pre-v2 reths fail loudly at start,
fallback is --no-slowdisk. Restores/clones of reth nodes to SLOWDISK=True
hosts now offload statics automatically (NVMe is the scarce resource).
Note: reth also has --datadir.static-files for a flag-based layout later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Candidate fix for the receipts RLP decode failure that blocks snap sync
past block 4,178,637 (2026-04-12): peers dropped with 'invalid message
(code 10) receipt 1: rlp: ListEnd not positioned at EOL' on the previous
newest published image.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
