The reload is in-process and takes 10-30 ms; the fixed sleep was the window in
which a removed upstream was already unrouted in dshackle while the gateway still
held the old announcements and kept sending queries it could no longer answer.
Poll the log for 'Reloading config has been completed' (or the fail/drop
markers), SETTLE stays the upper bound. Operator 2026-09-19.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KjqejYycVyVvrxV9aTDhgX
Operator ruling: NEVER force a reconnect on a routine reload/rpc-update run.
Only upstream REMOVALS or new disabled-method ADDITIONS require a gateway
reconnect to be seen by dRPC gateways.
Implement snapshot-based comparison:
- Before SIGHUP, read current advertised set (upstream IDs + disabled methods)
- Compare against previous snapshot (if exists)
- Drop established gRPC connections ONLY IF:
* an upstream ID disappeared, OR
* a new disabled-method entry appeared
- Pure additions or no change: reload only, no drop
- First run with no prior snapshot: do NOT drop, just write snapshot
- Always rewrite snapshot after successful reload
- All operations best-effort (failures never fail the script)
Live-measured: upstream/method additions propagate over existing connection
in ~16s (no reconnect needed).
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Extend reload_dshackle.sh to drop established gateway gRPC connections on
port 2449 after a successful SIGHUP config reload, forcing the dRPC edge
to reconnect within ~1s and re-read the advertised chain/method list.
Gateways only re-initialize advertisements on reconnection (verified live:
0.03-0.06s serving gap, 33 re-probes in 4s). This allows deploy-time upstream
additions (new chains like mova, new backends like us-50) to be seen by
dRPC immediately rather than waiting for natural reconnection.
The TCP-level drop is best-effort: failure does NOT fail the script.
Operator ruling: this stays a TCP-level drop, NOT a dshackle code change.
Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
dshackle's SIGHUP reload silently fails for upstream REMOVALS (and method-set
changes, same remove+add path): it logs "Config is not reloaded, cause - ...",
updates the in-memory config but leaves runtime selectors stale (drift), and keeps
serving the old config. No runtime admin API exists to mutate upstreams, so a
container restart is the only way to actually apply such changes.
Replace the blind SIGHUP with: per-container SIGHUP, then check post-SIGHUP logs
for the failure marker ("Config is not reloaded, cause -") or a dropped concurrent
HUP ("Reloading is in progress"); on either, docker restart to load config fresh.
"Reloading config has been completed" is NOT success (finally-block) — deliberately
not treated as such. Exit 0 when config ends up applied (via SIGHUP or restart);
non-zero only if a needed restart fails or the container doesn't come back.
Layered on top of the override-weave (apply-dshackle-overrides.py) and the 1:1
invariant gate (validate-dshackle-1to1.py) — both run unchanged before the reload.
Co-Authored-By: Claude <noreply@anthropic.com>
Insert validate-dshackle-1to1.py between the overrides weave and the HUP signal:
a dshackle config routing >1 node for the same chain is REFUSED (exit 1) so the
live config stays active. Structural enforcement of 1 dshackle = 1 node/chain —
we can't attribute traffic to multiple nodes behind one proxy (no per-upstream
request metric; conn-seconds biases it). See /root/proxy-1to1-invariant-plan.md.
Co-Authored-By: Claude <noreply@anthropic.com>
/root/rpc-local/dshackle-overrides.yaml (host-local, like rpc-local/compose.d) is
re-woven into main_configs/*.yaml by reload_dshackle.sh on every config push.
Method-disable decisions are local to each gateway's market (operator eviction
rule: cost_ratio = time_share/call_share at flat pay); this makes them durable
inputs instead of fragile output patches. Step 1 of moving dshackle config
generation onto the gateway hosts (step 2: ssh forced-command upstreams fetch,
generic dshackle.yaml include list derived from drpc-gateways.json).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
