Disaster Recovery Planning

Backups 101 established the habit of backing up and restoring a single machine. Proxmox Backups extended that to VM and container snapshots. This page is about the part most homelabs never get to: could you actually rebuild the whole thing - cluster, networking, secrets, and all - if it were gone tomorrow?

Why a cluster needs more planning than one box

A single-machine homelab has one obvious recovery story: restore the backup onto new hardware. A multi-node Proxmox cluster with IaC, GitOps, secrets management, and a dedicated firewall has several interdependent pieces, and losing all of it at once (fire, theft, a serious ransomware incident) means rebuilding an ordered sequence, not just restoring one disk image. Without a written plan, that sequence lives only in your head - which is exactly the resource least available during an actual disaster.

Document the rebuild steps, not just the backups

Write down, somewhere durable and outside the infrastructure it describes (not solely in a note on the Proxmox host itself), the actual order of operations:

1. Reinstall Proxmox VE on new hardware (version X, from USB).
2. Restore age/SOPS private key from password manager (see
   Secrets Management) - needed before anything else decrypts.
3. Clone infra repos (Terraform, Ansible, compose) from Git remote.
4. Re-run `tofu apply` to recreate VMs.
5. Re-run Ansible playbooks to configure them.
6. Restore Proxmox Backup Server data / restic repo onto new storage.
7. Rejoin cluster nodes (pvecm add) once quorum hardware is back.
8. Restore firewall (pfSense/OPNsense) config from its own backup.
9. Verify each service, starting with DNS and the reverse proxy.

The value isn't the specific list - it's that a list exists at all, gets updated when your setup changes, and lives somewhere you can read it without the infrastructure it's describing being available (a printed copy, or a note in a password manager, not a wiki page hosted on the cluster itself).

Test the restore, for real, at cluster scale

Backups 101 made the point that an untested backup is a theory. At this scale, testing means more than restoring one file - actually rebuild a VM from your Proxmox Backup Server or restic repo onto a scratch node or spare drive periodically, and separately, actually run through the IaC + Ansible rebuild sequence against a throwaway VM to confirm the automation still works end to end. Tooling drifts - a Terraform provider update, an Ansible collection deprecation, a changed Proxmox API - and a rebuild procedure that worked a year ago can quietly stop working without you noticing until the day you need it.

⚠️ Risk: the most common failure mode here isn't a bad backup - it's a rebuild procedure that assumes a piece of context (a manual step you did once and forgot to document, an environment variable set on your laptop) that only lives in your head. A restore test that involves you improvising from memory hasn't actually validated the written plan.

Off-site backups for the whole cluster, not one machine

Scale the 3-2-1 approach (see 3-2-1 backup rule) to cover everything that matters:

• VM/container backups - Proxmox Backup Server data, replicated or synced off-site, not just kept on a local PBS instance in the same building as the cluster it's protecting.
• Config repos - Terraform, Ansible, compose, and GitOps repos pushed to a remote you don't control the only copy of (a hosted Git service, or a second self-hosted Git instance genuinely off-site).
• The secrets key - your SOPS/age private key, backed up somewhere separate from both the repos it decrypts and the cluster it configures (see Secrets Management).
• Firewall config - pfSense/OPNsense configuration exports, backed up on their own schedule; rebuilding network rules from memory after a total loss is exactly the kind of thing a written plan should cover instead.

Losing a single node is a Tuesday with a cluster in place. Losing the building is the scenario this section is actually for - plan for it while it's hypothetical, not while it's happening.

Next: Observability at Scale.