Proxmox

Proxmox VE is the homelab hypervisor. It runs a single VM: OPNsense, which handles routing and firewalling for the entire network.

Proxmox itself is managed with Ansible from the proxmox repo.

Ansible

The Ansible setup covers thirteen playbooks — nine for the hypervisor itself, and four that reach past it into the OPNsense guest over its API:

How the three backup mechanisms fit together

Four of the hypervisor playbooks are about backup, and they cover deliberately different things — vzdump only ever backs up guests, which would leave the hypervisor's own configuration unprotected if that were the whole story:

graph LR
    subgraph host[Proxmox host]
        guests[Guests<br/>OPNsense VM]
        cfg["Host config<br/>/etc/pve, network, apt"]
    end

    guests -->|vzdump job<br/>nightly 21:00| nas[(Synology<br/>NFS share)]
    cfg -->|systemd timer<br/>nightly 22:00| nas
    nas -->|restore_test.yml<br/>on demand| test[Throwaway VMID<br/>network isolated]
    nas -->|Hyper Backup| b2[(Backblaze B2<br/>off-site)]

The vzdump job runs at 21:00 in snapshot mode with zstd compression and a 7 daily / 4 weekly / 3 monthly / 1 yearly retention policy, and it's managed through the Proxmox API rather than by editing jobs.cfg by hand. The host-config archive runs an hour later from a self-contained script and its own systemd timer, so it keeps working whether or not anyone runs Ansible. Both land on the same NAS share, which Hyper Backup then replicates off-site — the same 3-2-1 arrangement used for the servers.

restore_test.yml closes the loop by proving the guest backups actually restore, into a throwaway VMID with the network isolated so it can never collide with the running guest.

Knowing whether the backup ran

vzdump has a mailnotification setting, and relying on it here would have been a silent failure. Mail does not leave this host: the internal resolver answers authoritatively for the domain without an MX record, so postfix simply bounces. The notification path was configured, looked configured, and delivered nothing — the classic shape of a backup you believe is being watched.

monitor_vzdump.yml replaces it with metrics instead of mail. A small script reads vzdump task results from the PVE API every fifteen minutes and writes per-guest series to the Node Exporter textfile collector:

• pve_vzdump_success — did the last run of this guest's backup succeed
• pve_vzdump_last_success_timestamp_seconds — when it last succeeded, which is what catches a job that stopped running rather than one that failed
• pve_vzdump_last_duration_seconds — how long it took, so a job that quietly starts backing up nothing is visible as a collapse in duration

Alerting then lives in Prometheus alongside every other rule, rather than in a mail path that nothing verifies. The playbook refuses to install if the textfile directory is missing, and asserts afterwards that the generated file really contains per-guest rows — a metrics file that exists but is empty would look exactly like a healthy one to a rule that only checks for failures.

This closes the same gap on the hypervisor that the Borgmatic deadman switch closes on the servers: the question is never "did the backup fail", it's "would I find out if it stopped happening at all".

Refusing to overwrite what it doesn't manage

configure_backups.yml templates the entire /etc/pve/jobs.cfg, which creates an obvious hazard: any vzdump or replication job added through the Proxmox GUI would be silently erased on the next Ansible run. Declarative management of a file you don't exclusively own destroys whatever else is in it.

Rather than merging — which means parsing and reconciling a format Proxmox owns, and failing in more interesting ways than it solves — the playbook reads the current file first, collects every job block header, and aborts if it finds one it doesn't manage:

/etc/pve/jobs.cfg contains job blocks this playbook does not manage: [...].
The template would overwrite them. Add the jobs to this playbook, or remove
them before running.

The same reasoning applies wherever a declarative tool here would silently destroy out-of-band state: the playbook is made to stop, not to be made quieter. An automation that guesses is worse than one that refuses, because the guess is invisible and the refusal isn't.

All playbooks are run from the ansible/ directory:

# Base configuration (DNS, timezone)
ansible-playbook configure_proxmox.yml

# Backup job
ansible-playbook configure_backups.yml

# Full system update
ansible-playbook update_proxmox.yml

# Install/update Node Exporter
ansible-playbook install_node_exporter_community.yml

Update safety

The update playbook guards against accidental major version upgrades. It checks whether next-release repositories (e.g. trixie on a bookworm system) are present in /etc/apt/ and aborts if so. A major upgrade requires explicitly setting allow_major_upgrade: true.

It can also take a root snapshot before upgrading (take_pre_upgrade_snapshot=true, off by default), which turns a bad upgrade from a restore into a rollback. The playbook detects the backend from the root mount rather than assuming one:

That third row is the interesting one, and it's there because the original implementation assumed ZFS. This host is actually LVM (pve/root on ext4), where zfs answers cannot open 'rpool': dataset does not exist — a string the old error handling didn't recognise, so instead of skipping the snapshot it took down the whole upgrade run. The fix could have been to skip quietly when ZFS isn't there. It deliberately isn't: a silent skip hands you false confidence at the exact moment you asked for a restore point, so an unrecognised backend aborts with an actionable message instead.

The retention difference isn't arbitrary either. LVM snapshots have a fixed copy-on-write area and are invalidated if it fills, and each live snapshot slows writes to root — so hoarding them costs performance and buys nothing. Peak volume-group usage is (keep + 1) × size, because the new snapshot is created before the old ones are pruned, and a pre-flight check fails with the actual free space in the message rather than letting lvcreate fail with "insufficient free space". Restoring from one is lvconvert --merge plus a reboot, not an in-place rollback like ZFS.

Reboot safety

The playbook reboots automatically if the kernel changed, and then waits for the guests to come back, which matters more here than it would elsewhere: one of those guests is the router the rest of the network depends on. Getting the reboot itself to work took considerably more than calling reboot.

The host is a CWWK CW-AD4L-N mini-PC (AMI firmware 5.27), and its firmware does not re-enumerate boot devices on a warm reset — which is exactly what shutdown -r now performs. The machine came back into BIOS setup, and setup's "Save & Exit" warm-resets straight back into setup. Only pulling the power recovers it. On a host whose single VM is the router, that means the entire network stays down until someone is physically present.

There were two independent causes, and fixing either alone would have left the trap armed:

1. No active EFI boot entry of its own. The only active entry was the firmware's auto-generated UEFI OS fallback, which is created by a disk scan — and a warm reset skips the scan. The disk's own proxmox entry existed but had its active flag unset, so there was nothing left to boot. Recreated with efibootmgr -c.
2. The kernel used the default ACPI reset. reboot=efi on the kernel command line makes it call the firmware's own cold-reset service instead.

Both are now guarded by tasks that run immediately before the reboot, and the guard fails the run if BootOrder contains no active proxmox entry rather than rebooting into an unbootable machine. Two details from that are worth keeping:

• The check must run before the thing it protects. Rewriting GRUB from an Ansible handler would have applied it at the end of the play — after the reboot it was meant to make safe. The same goes for verifying boot entries afterwards: a post-reboot check on a host that didn't come back has nobody to report to.
• The guard asserts the right property. An early version required the proxmox entry to be first in BootOrder, and consequently failed every run: this firmware re-promotes its own fallback entry to the head on every boot. Both entries point at the same partition and both are active, so either one boots. What actually matters is that the disk's own entry exists, is active, and is in the boot order at all — the firmware's fallback is the one that can't be relied on, because it only exists after a scan that warm resets don't do.

Verified with a real reboot: 63 seconds to return, reboot=efi present in /proc/cmdline, the router guest and all PVE services back up. One trick made that test cheaper — reboot=efi only takes effect from the next boot, so the reset type was set on the already-running kernel first (echo efi > /sys/kernel/reboot/type) to validate the change without spending an extra reboot on it.

Host exposure

The host itself has a small listening surface: SSH, the Proxmox web UI, and Node Exporter on port 9100 bound to a single interface. SSH is key-only and the ansible account's key carries a from= restriction limiting it to the LAN. Outbound, the host mounts an NFS share on the NAS, which is where backups land.

pve-firewall is deliberately not enabled, and no playbook manages it — the reasoning, and the condition under which that would change, is on the Architecture decisions page.

My experience

I had used Proxmox before for running LXC containers and virtual machines, so it was a familiar choice. The idea to run OPNsense as a VM inside Proxmox came from a ServeTheHome article. The best part of this setup is being able to snapshot the VM before a major OPNsense upgrade — if something breaks, rolling back is instant.

Managing Proxmox with Ansible has been smooth, though the automation is still partly a work in progress.