Commit Graph

9 Commits

Author SHA1 Message Date
d896378eb7 Add optional/ephemeral volume notation for restore resilience
Introduce x-optional-volumes for beacon sidecars (prysm, nimbus, lighthouse,
teku, lodestar) that are checkpoint-sync capable and rebuild state quickly.
Mark reth sockets volumes as ephemeral (runtime IPC artifacts).

Script changes:
- volume-utils.sh: add is_optional_volume_key() function
- restore-volumes.sh: tolerate missing optional/ephemeral volume archives
- clone-backup.sh: skip missing optional/ephemeral volumes gracefully
- list-restorable.sh: only require archives for non-optional persistent volumes

This fixes the de-25 ethereum prysm incident (entire 1.1T restore failed
due to missing prysm archive) and arc-testnet reth sockets restore loops.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-08-04 06:10:55 +00:00
rob
95360bd238 fix: rewrite tar extract onto target volumes root when it is a cross-device symlink
Hosts like rpc-us-32 put /var/lib/docker (or volumes/) on another mount;
tar -C / then EXDEV on every var/lib/docker/volumes/... member. Canonicalize
the deepest static prefix (readlink -f /var/lib/docker/volumes) and append a
--transform after SLOWDISK_TRANSFORMS. Keying only on /var/lib/docker misses
a real docker root with volumes/ as the symlink. No-op on normal hosts.
Per-volume _data static-file offload stays on SLOWDISK_TRANSFORMS.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-19 02:24:56 +00:00
rob
1ebabee93d volume-utils: per-volume-kind restorable floor (_config tars are legitimately tiny)
The 1MB near-empty guard (added for purged-volume zstd stubs) rejected every
legitimate _config volume archive (cronos: 68KB), so any node whose registry
entry includes a _config volume reported Restorable: 0 despite healthy
multi-GB data backups (cronos had three; 14 registry entries carry _config
volumes). _config archives now pass at >=4KB; data volumes keep the 1MB floor.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-08 12:01:44 +00:00
e571efbea1 fix(backup): read remote backup list from stdin in selector
select_newest_remote_backup_from_list was sorting an empty process
substitution instead of the piped list-backups.sh output.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 04:38:31 +00:00
58adc446e2 backup: harden lifecycle fencing and skip near-empty archives
backup-node.sh now stops services, fences COMPOSE_FILE during backup, and
restores via EXIT trap; consumers skip <1MB .tar.zst artifacts so purged-
volume junk cannot shadow real backups.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-03 04:38:00 +00:00
345538954d restore/cleanup: implement static-file -> /slowdisk offload + free it on removal
restore-volumes.sh: pre-create static-file symlinks from the backup's .txt manifest so
the immutable ancient/freezer dirs land on /slowdisk (SSD) and extract THROUGH the
symlinks via tar --keep-directory-symlink (was --dereference, which clobbered them);
hot state stays on the primary disk. Cleans stale /slowdisk targets first (no leak on
re-restore). Safe fallbacks: no /slowdisk / no manifest / no static paths -> normal
extract. Reth excluded (reth dropped whole-dir static-file symlinks).

volume-utils.sh: add delete_slowdisk_targets_for_key() — follows a volume's symlinks and
sweeps the rpc_<key>__data_ pattern under /slowdisk (matches delete-volumes.sh).

cleanup-volumes.sh: free the /slowdisk static data before docker volume rm (was leaking),
and fix the fragile substring used/unused match to an exact name match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 03:05:04 +00:00
goldsquid
aa5ffb63b6 fix 2026-06-06 11:49:31 +07:00
goldsquid
1aef140a9a if the backup storage is local do not use webdav 2026-06-06 11:36:58 +07:00
goldsquid
91ef991773 make the volume scripts ignore ephemeral volumes 2026-06-06 09:57:48 +07:00