The doma --filter run regenerated this global aggregate from only the
doma subset, emptying it. Restore the complete file; doma resolves via
its dshackle label so no reference-endpoint entry is required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conduit OP-stack L2 on Ethereum mainnet L1. Canyon-at-genesis; op-geth
ingests the Conduit genesis.json directly (no op-reth hash pitfall).
Config served from Conduit slug doma-mainnet-qvzsfv8nv0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
find -type d skipped static dirs that are already offloaded (a symlink to /slowdisk is type
l, not d) — so show-static-file-size.sh reported zero static for an offloaded node (e.g. bob
on de-35), and backup-node.sh would drop them from the manifest (breaking re-offload on the
next restore). Match dirs AND symlinks now (root-level entries too).
show-static-file-size.sh also tags each static dir with its location:
[OFFLOADED -> /slowdisk/...], [on-disk], or [BROKEN SYMLINK ...].
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SLOWDISK value is emitted by Python templates as capitalized booleans (True/False), so
match "True" (also accept manual lowercase "true"); anything else = offload off. --no-slowdisk
sets SLOWDISK=False to override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Use the single SLOWDISK env var from .env (already set to "true" on the dedicated-extra-disk
hosts, e.g. us-35). Offload runs only when SLOWDISK is exactly the literal "true" (case
matters) and the --no-slowdisk flag was not passed. --no-slowdisk forces it off even when
SLOWDISK=true (extra disk full). Replaces the prior NO_SLOWDISK inversion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Static-file offload gating, clearer semantics:
- NO_SLOWDISK defaults to TRUE (offload OFF) — safe default everywhere.
- A host with a real dedicated extra disk at /slowdisk sets NO_SLOWDISK=false in its .env
to ENABLE the offload.
- New --no-slowdisk CLI flag forces NO_SLOWDISK=true, overriding the .env false — for when
the extra disk exists but is full. Flag is parsed position-independently; the positional
args ($1 compose, $2 remote source) are preserved.
Offload runs only when NO_SLOWDISK is false AND the flag was not passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A `/slowdisk` directory exists on hosts even when it is just a folder on the root disk
(no dedicated extra disk) — offloading there gives no benefit. Source the host .env and
require SLOWDISK to be set (operator sets it only on hosts with a real extra disk mounted
at /slowdisk) before activating the static-file -> /slowdisk symlink offload. Unset =
normal extract everywhere. Target path stays /slowdisk (the fixed in-container mount).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Verified static-file dirs across the reachable fleet via per-host cursor inspection
(de-13/14/16/22/27/30/31/32/35, us-16/41, uk-4). Findings:
- nitro freezers are network-prefixed (arbitrum-one/nitro/l2chaindata/ancient ... up to
1.3 TB), so the old chain/ and data/ prefixes matched nothing;
- missing: snapshots (erigon3/op-erigon/cosmos), lightchaindata/ancient, nested l2geth
geth/geth/chaindata/ancient, aztec archiver, nitro classic-msg/ancient;
- bare `ancient` matched nothing (all are nested).
List rewritten to canonical entries. Matching (backup-node.sh manifest + show-static-
file-size.sh) now: entry with NO slash = root-level only (so `snapshots` does NOT catch
postgres pg_logical/snapshots), entry WITH a slash = path-suffix via find -path "*/X"
(matches any prefix: network-prefixed nitro, nested datadirs). The manifest now records
the CONCRETE per-volume path so restore-volumes.sh recreates the exact symlink.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
restore-volumes.sh: pre-create static-file symlinks from the backup's .txt manifest so
the immutable ancient/freezer dirs land on /slowdisk (SSD) and extract THROUGH the
symlinks via tar --keep-directory-symlink (was --dereference, which clobbered them);
hot state stays on the primary disk. Cleans stale /slowdisk targets first (no leak on
re-restore). Safe fallbacks: no /slowdisk / no manifest / no static paths -> normal
extract. Reth excluded (reth dropped whole-dir static-file symlinks).
volume-utils.sh: add delete_slowdisk_targets_for_key() — follows a volume's symlinks and
sweeps the rpc_<key>__data_ pattern under /slowdisk (matches delete-volumes.sh).
cleanup-volumes.sh: free the /slowdisk static data before docker volume rm (was leaking),
and fix the fragile substring used/unused match to an exact name match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- op-geth: drop stray trailing ` \` on --hvm.genesisheader and
--tbc.leveldbhome (the backslash was decoded as part of the hVM
genesis header -> "invalid byte U+0020 ' '" crash)
- op-node: add OP_NODE_OVERRIDE_ECOTONE/CANYON/DELTA=1725868497
(rollup.json carries only regolith_time; without these the later
fjord override tripped "fork fjord set but prior fork ecotone missing")
- op-node: OP_NODE_BSS_WS no longer wrapped in literal quotes
- bump to Hemi's current pins (mainnet is past isthmus/holocene):
op-geth 05d4d8f, op-node 7c70d2d
Values mirror hemilabs/hemi-node mainnet scripts/gen.sh + config.json.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consensus service renamed arc-testnet-consensus -> arc-testnet-node: it is now
generated by templates/nodes/malachite.yml (node: malachite) instead of being
hacked into the arc client template's indexer block. Container body unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
op-node's default engine-API RPC timeout is short; on nodes whose EL responds
slowly to engine_newPayload/forkChoiceUpdated (e.g. boba-mainnet op-reth pruned
during catch-up) op-node times out driving the EL and the execution head freezes
while op-node keeps deriving from L1. Raising the engine RPC timeout to 120s
prevents this class of stall fleet-wide.
Regenerated from env/op/node.env: adds the env var to every op-node service
(116 geth + 40 reth + 22 erigon composes); additions only, no other changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
genesis.json had no jovianTime, so op-geth/op-reth never activated Jovian
and op-node rejected Jovian-era L1-info txs ("unexpected length"). Added
jovianTime=1772548201 to the genesis config (EL source of truth for both
geth and reth). Removed OP_NODE_OVERRIDE_JOVIAN from the op-node env: the
local rollup.json already carries jovian_time, making the override redundant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
op-reth rejects geth's --nodiscover flag, which the op-geth base template
emits for node_sync nodes. Use reth's --disable-discovery instead.
- hashkeychain-testnet: now node_sync/consensus-layer (was execution-layer
with an empty DB, which could not backfill) + --disable-discovery
- xlayer mainnet/testnet (archive+pruned): --nodiscover -> --disable-discovery
(these were latently broken on restart)
Fix lives in the op-reth.yml base template so all reth nodes are covered;
the redundant override in op-reth.boba.yml was removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chain 53302 is legacy Canyon-at-genesis (block 0 has no Shanghai
withdrawalsRoot). Upstream op-reth always injects it and computes the wrong
genesis hash 0xfa6f21ae vs canonical 0x7274a90e, so it crash-loops against
the DB. conduit-op-reth has legacy-Canyon genesis compatibility for 53302.
Regenerated from context.yml (reth client_image override) + op-reth.yml
template (historicalrpc for superseed-sepolia fresh-sync backfill).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conduit's downloaded genesis.json/rollup.json are faulty (rollup.json
lacks chain_op_config + post-Fjord fork times, crashing op-node v1.19.0).
The fixed configs live in op/bob/sepolia/. Removed the client_genesis /
node_rollup_json Conduit URLs from context.yml and regenerated, so all 7
bob-sepolia composes (geth + reth) now bind-mount ./op/bob/sepolia:/config
and drop the curl-from-Conduit init containers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
op-node was crashing post-Jovian and the rollup/genesis configs were missing
two fields present in the live sequencer (sepolia.superseed.xyz
optimism_rollupConfig):
- rollup.json: add pectra_blob_schedule_time=1742486400 (L1 Sepolia Pectra)
- genesis.json: eip1559DenominatorCanyon 0 -> 250 (match chain_op_config)
Verified against the live sequencer. Fixes op-node 'unexpected length: 178'
L1-info parse error. karst_time not yet published by the sequencer (Karst
Sepolia ~2026-06-17 16:00 UTC) - to be added once available.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conduit's rollup.json lacked chain_op_config and fork times required by op-node v1.19.0, causing a crash loop. Mount op/bob/sepolia locally (genesis + rollup) and drop Conduit init containers.
Co-authored-by: Cursor <cursoragent@cursor.com>
bob-sepolia is past the Jovian hardfork (jovian_time=1772548201). Conduit's
downloaded rollup.json omits jovian_time, so op-node misparsed the 178-byte
Jovian L1-info deposit txs ("data is unexpected length: 178") and the node
stalled at genesis. Add OP_NODE_OVERRIDE_JOVIAN to all bob-sepolia variants.
Remove op/bob.by: an unused local genesis/rollup backup. Source of truth is
the genesis/rollup downloaded from the Conduit URLs in context.yml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Circle's arc-execution/arc-consensus images are distroless with only the
`arc` user and no `root` entry in /etc/passwd, so `user: root` failed with
"unable to find user root". Run by numeric uid 0:0 instead (no passwd
lookup needed; root can write the root-owned named volumes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- superseed-sepolia: add holocene/isthmus/jovian fork times to rollup.json (CL)
and genesis.json (EL). jovian_time=1768978801 is superseed's chain-specific
activation (verified live via sepolia.superseed.xyz optimism_rollupConfig);
the official superseed-xyz/node repo + the old committed config both lacked it,
causing op-node "failed to parse L1 info deposit tx ... unexpected length: 178".
- op-reth v2.3.0 -> v2.3.1 across all op-reth composes (Karst-mandatory release,
Sepolia activation Jun 17 2026 16:00 UTC).
- op-sepolia: --chain=optimismsepolia -> optimism-sepolia (invalid chain spec
prevented op-reth from starting).
- soneium-minato-sepolia (reth/geth/erigon): L1 upstream vars now resolve to
ETHEREUM_SEPOLIA_* (were ETHEREUM_MINATO-SEPOLIA_*, nonexistent -> no L1 RPC);
state-cache shell var de-hyphenated; pectra blob-schedule override now emitted.
NOTE: superseed karst_time not yet published by the chain; must be added before
Jun 17 2026 once superseed announces it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- sync-status.sh: resolve protocol family from the compose x-upstreams chain
label via the registry (reference-rpc-endpoint.sh --protocol) instead of
path-substring guessing; legacy path detection kept as fallback for
composes without a resolved label. Unknown families report
'unsupported protocol: X' honestly instead of a bogus eth_chainId error.
Aztec reference lookup falls back to slug urls when rollup_version is
not in the registry.
- reference-rpc-endpoint.sh: new --chain <slug> (urls by registry key,
works for idless non-EVM entries) and --protocol <slug> modes; existing
chainid and --rollup-version lookups unchanged.
- mantle-sepolia: chainid 5001 -> 5003 (verified live: 0x138b), label and
--networkid now correct
- metis-sepolia: label resolves via registry override (drpc chains.yaml
carries wrong id 59901; live chain is 59902, verified via official RPC)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aztec-testnet, solana, ripple x-upstreams now carry their drpc slug as
chain label (was empty -> dshackle ignored the upstreams)
- reference-rpc-endpoint.json: every chain now has a protocol field
(eth/bitcoin/solana/ripple/aztec/...) for protocol-family dispatch in
scripts, plus https://{slug}.drpc.org appended as LAST fallback url
(independent endpoints stay preferred; overrides win entirely)
- registry grew 247 -> 333 entries (all drpc protocols imported, not just eth)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Aztec testnet rollup was redeployed on Sepolia; 3.0.2 computes the old
genesis archive root and crash-loops. 4.3.1 matches the deployed rollup
and bootstraps from snapshot. Verified live on rpc-de-13.
Includes regen drift from current templates for other stale compose files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The old script grouped backups with cut -d'-' -f1-3, which mis-grouped
volume keys and permanently deleted ~360 archives on 2026-06-11. Replace
with safe retention: group by full volume key regex, keep 3 newest per key,
move excess to /backup/trash, and protect uploading-*, broken_*, and
deprecated-* prefixes.
Co-authored-by: Cursor <cursoragent@cursor.com>
Ronin migrated from standalone sidechain to OP-stack L2 operated by
Conduit, with EigenDA V2 for data availability. New composes follow
conduitxyz/ronin-migration-reth-docker: conduit-op-reth fork +
op-node v1.16.5 + eigenda-proxy 2.7.0 relay.
- op/ronin/{mainnet,saigon}: Conduit genesis (bedrockBlock patched to
55577500 / 45528550) and rollup.json (alt_da GenericCommitment injected)
- saigon is a NEW chain post-migration: chainid 202601, L1 sepolia
- also regenerated: xlayer-testnet composes (dshackle chain label was
corrupted to 'ronin' by an override typo), native ronin/ronin composes
(upstream labels follow new chain ids), reference-rpc-endpoint.json
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Superchain Sepolia hardfork ~2026-06-19 drops op-geth support; op-erigon is
already broken post-hardfork. op-reth is the migration target for Superchain
members. Generates archive-trace, minimal-trace, and pruned-trace variants
via the 'full_minimal_trace' profile (mirroring base-sepolia / op-sepolia).
Pre-existing template change: bind-mount './op/celo/sepolia:/config' migrated to
parameterizable named docker volumes (CELO_SEPOLIA_OP_GETH_*_CONFIG /
CELO_SEPOLIA_OP_GETH_*_NODE_CONFIG with sensible defaults). Surfaced when
running update.sh on top of the current rpc/ submodule HEAD.
The rskj compose has an nginx proxy wrapper in front of the rskj client.
The wrapper listens on port 80 and proxies to rskj:8545 internally (this is
required because rskj's hosts whitelist is hard-coded to localhost only,
so traefik forwards must look local to rskj). Traefik labels were pointing
to port 8545, hitting the rskj client directly, which rejected the request
because the Host header didn't match localhost.
Fix: traefik loadbalancer.server.port now 80 for both rootstock-mainnet
and rootstock-bamboo (mirrors the parent rpc-client.yml template's
existing client_proxy_required logic, which the rskj client template
inadvertently dropped).
Verified on de-31: rootstock-mainnet eth_blockNumber returns block
8,872,189 after the fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These flags were removed in Nitro v3.10 and cause Fatal config errors on
startup. Affected chains use AnyTrust DAS via REST aggregator without a
separate relay (orbit-no-relay branch):
- alephzero (mainnet+sepolia)
- connext-sepolia
- everclear-mainnet
- opencampuscodex-sepolia
- playblock-mainnet
- plume (mainnet+testnet)
- real-mainnet
The parent-chain node URL and sequencer-inbox-address are now auto-derived
from --parent-chain.connection.url. The --node.da-provider.* flag family is
gone entirely.
Verified live on rpc-de-13 (everclear): container Up, REST aggregator URL
accepted, head block loaded, RPC listening.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
${?DOMAIN} substitution caused rskj to crash with 'Can't read config'.
Hardcoding the host-specific domain for now (rootstock only runs on
de-31). For multi-host deployment later, would need a per-host
templated node.conf or a proxy that rewrites Host header.
rskj's hosts whitelist is exact-match (no wildcards). Use HOCON env-var
substitution ${?DOMAIN} to inject the per-host public domain into the
allowed list. Pass DOMAIN env var into the container via the rskj
template.
Allowed hosts list: [localhost, 127.0.0.1, ::1, ${?DOMAIN}]
- localhost variants for direct/internal access
- DOMAIN for traefik-forwarded requests (Host header = public domain)
Setting hosts via -Dhosts.0=*, -Dhosts.1=*, etc creates an OBJECT in
HOCON (indexed entries), but rskj's schema expects LIST. Result: rskj
warns and falls back to default whitelist (localhost only).
Drop the hosts.X JVM properties entirely. The file /etc/rsk/node.conf
already declares hosts = ["*"] which IS a HOCON LIST. Without the JVM
override, the file value wins.
The rsksmart/rskj:VETIVER-9.0.1 image bakes an env var:
RSKJ_SYS_PROPS=-Drpc.providers.web.http.bind_address=0.0.0.0
-Drpc.providers.web.http.hosts.0=localhost
-Drpc.providers.web.http.hosts.1=127.0.0.1
-Drpc.providers.web.http.hosts.2=::1
These JVM -D system properties take precedence over /etc/rsk/node.conf
in HOCON, so the Host whitelist always resolves to {localhost,127.0.0.1,::1}.
Result: traefik routing to rskj at IP rootstock-mainnet-client:8545
arrives with Host header that doesn't match those three. rskj returns
HTTP 400, traefik translates to 502 Bad Gateway.
Override the env var in the compose template so the http hosts whitelist
contains '*' (any host). Traefik's ipallowlist middleware is the actual
gatekeeper. Affects rootstock-mainnet AND rootstock-bamboo.
VETIVER-9.0.1 enforces rpc.providers.web.http.hosts as a whitelist;
empty list / default rejects everything except 'Host: localhost'.
The traefik route uses 'customrequestheaders.Host=localhost' middleware
to rewrite the Host header, but that middleware appears not to be
applied for the rootstock route - real requests still arrive at rskj
with Host: rpc-de-XX.stakesquid.eu and rskj returns 400 Bad Request,
which traefik translates to 502 Bad Gateway.
Wildcard whitelist is fine here because traefik (with ipallowlist
middleware) is the actual gatekeeper.
Confirmed empirically: 'wget --header=Host:localhost' returns valid
JSON-RPC, anything else returns 400.
VETIVER-9.0.1 silently ignores rpc.modules when configured as a list of
{name,version,enabled} objects. Switch to canonical OBJECT format from
rskj's expected.conf:
modules = { eth { version = "1.0", enabled = "true" } ... }
Without this, the RPC server starts but no modules are registered, so
eth_blockNumber returns method-not-found and show-status flags as 'error'
even though the chain itself imports blocks fine.
Also added 'hosts = []' under rpc.providers.web.http (LIST per
expected.conf, was missing). Same fix applied to bamboo testnet config.
Note: this only resolves mainnet. Bamboo still needs DB recreate due to
unrelated 'Invalid block header size: 22' corruption.
Required to fix zksync-era-mainnet on de-13 which is failing with
'migration 20250820140638 was previously applied but is missing in the
resolved migrations'. We were 15 minor versions behind the latest stable
GitHub release (core-v29.16.0). Cascades to abstract/lens external-node
composes which inherit the base version (none currently deployed).
maru v1.2.x strictly validates l2-eth-api-endpoint as a URL and crashes when SEQUENCER
env var is empty. Old maru tolerated empty. Fixes 4 stuck Linea instances on
de-27, de-32, us-40 — all besu/geth running but maru crash-looping with config error
'Required type class java.net.URL could not be decoded from a String value: '
1.7.3 introduced a regression where --tries-verify-mode=none silently overrides
--state.scheme=path → hash, causing 'Incompatible state scheme' crash on minimal
nodes built with path scheme. 1.7.2 was confirmed working before the auto-bump.
v1.2.0 tag doesn't exist on Docker Hub, causing compose build
failures on de-27, de-32, us-40. Using the actual tag format.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Update drpcorg/dshackle from 0.75.17 to 0.75.21
- Required for Tron network support
- Updates both drpc.yml and drpc-free.yml
Co-Authored-By: Claude Agent <claude@stakesquid.eu>
- Replace outdated bootnodes (34.89.39.114, 35.197.249.21) with official ones from Polygon docs
- New bootnodes: 34.89.255.109, 34.185.137.160, 34.89.119.250, 34.89.40.235
- Increase maxpeers from 50 to 500 per Polygon docs recommendations
- Affects: Bor and all Erigon3 variants (archive, minimal, pruned)
Reference: https://docs.polygon.technology/pos/reference/seed-and-bootnodes/
Co-Authored-By: Claude Agent <claude@stakesquid.eu>
Old bootnodes (34.89.39.114, 35.197.249.21) were down causing sync failure.
Updated to official bootnodes from Polygon documentation.
Co-Authored-By: Claude Agent <claude@stakesquid.eu>
- polygon-amoy: reassigned from port 12188 (collision with hashkeychain-mainnet)
to 12190 via salt+2 in generate_ports()
- zircuit-garfield/testnet: fix empty --networkid= (was {{ chain_id }},
now {{ chainid }})
Co-Authored-By: Claude Agent <claude@stakesquid.eu>
Snap sync causes a deadlock on fresh nodes: consensus sends a zero
finalized block hash at startup which geth rejects, crashing consensus.
Full sync processes blocks sequentially from genesis and avoids this.
Co-Authored-By: Claude Agent <claude@stakesquid.eu>
Replace dead itrocket hostname with official seed from bundled config.toml:
461b27a9d1530eb47f62fe010a8d3e5d43b6740c@34.82.252.10:26656
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix networkid from 16601 to 16602 (confirmed by official docs)
- Bump version 3.0.3 → 3.0.4
- Replace dead bootnode (8.218.88.60) with bundled geth-config.toml
- Replace dead consensus seed (8.218.88.60:26656) with itrocket seed
- Same fixes applied to aristotle for consistency
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Overrides default gascap (600000000) to 0 (no gas cap) in BSC template.
Core inherits from BSC template, so also updated.
Co-Authored-By: Claude Agent <claude@stakesquid.eu>
op-reth moved from ghcr.io/paradigmxyz/op-reth to
us-docker.pkg.dev/oplabs-tools-artifacts/images/op-reth
starting with v1.11.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zkevm.datastream-version, zkevm.get-logs-timeout, zkevm.get-logs-retries
are unknown/deprecated in v2.63.0-RC11.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
claudeandClaude Opus 4.8