Backup

Both servers — monitoring and kontti — are backed up with borgmatic running in a Podman container. Backups go to the Synology DS920+ over SSH using the Borg protocol.

Schedule

Two systemd timers control when borgmatic runs:

• borgmatic.timer — triggers a backup daily at 04:00. Persistent=true ensures a missed backup runs on the next boot.
• borgmatic-check.timer — runs a full Borg integrity check every Sunday at 10:00, with a random delay of up to 10 minutes to avoid a sharp load spike.

The check runs in a separate container (borgmatic-check) so it can run independently of the backup schedule.

Pre- and post-backup hooks

Before each backup, a pre-backup script stops any services that need to be quiesced and exports specified Podman volumes to tar archives in the backup source directory. After the backup completes, the post-backup script restarts the services and cleans up the exported files.

This ensures that databases and other stateful services are in a consistent state when Borg reads the data.

Monitoring integration

Borgmatic reports its status to Prometheus via Pushgateway after each run:

# On success
echo "borgmatic_last_run_status 0" | curl --netrc --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>
echo "borgmatic_last_run_timestamp_seconds $(date +%s)" | curl --netrc --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>

# On failure
echo "borgmatic_last_run_status 1" | curl --netrc --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>

The backup.rules.yml alert rule fires if the last successful backup timestamp is too old, catching cases where the backup silently failed or the timer didn't run.

Authenticating the Pushgateway

--netrc is doing real work in those commands. Pushgateway used to accept unauthenticated writes — and unauthenticated deletes — from anything on the LAN, which made the entire backup deadman switch forgeable. Anyone with a foothold in a container could push a false borgmatic_last_run_status 0, or delete the metric group outright, and hide a genuine backup outage. That is precisely the signal these pages lean on hardest.

Two constraints shaped the fix:

• The port cannot simply be closed. kontti pushes to the Pushgateway on the other host, so binding it to localhost would break the cross-host deadman switch — the same one whose silent failures the rest of this page documents. Basic auth via Pushgateway's --web.config.file closes the hole without closing the path.
• The credentials must not appear on a command line. Borgmatic's hooks are shell commands, and command text ends up in the journal — which Alloy ships to Loki. curl -u user:pass would have written the password into the log pipeline. --netrc reads it from a root-owned 0600 file instead, mounted read-only into the borgmatic containers — so only a file path ever appears in argv. It's the same reasoning already applied to the Borg passphrase, which is read out of the config file rather than passed as an argument.

The bcrypt hash lives in the Ansible vault as a fixed value rather than being generated at template time: bcrypt's salt is random, so generating it on every run would render a different file each time and restart Pushgateway on every deploy. Changing the password means updating both vault values together.

One side effect was worth recording: --web.config.file puts /-/healthy behind auth too, so the container's old health check started failing on a service that was working correctly. The quadlet is world-readable and the container runs as 65534, which cannot read root's netrc — so the check now accepts both 200 and 401. Either answer proves the HTTP server is up, and 401 additionally proves the auth config loaded. A crashed process produces neither.

What this does not solve: the push is still plain HTTP, so the credentials cross the LAN in cleartext on every run. This closes off forgery — the threat that mattered, since any container could previously write or delete these metrics — but not interception. The bar moves from "anything on the network" to "on-path, or able to read a root-owned 0600 file". The same --web.config.file supports TLS, so the remaining step is certificate handling for an internal service rather than a design change; it is a known gap, not an oversight.

Prometheus data exclusions

When backing up the monitoring server, the Prometheus data directory is included but the WAL (write-ahead log) and chunks_head directories are excluded:

exclude_patterns:
  - /mnt/prometheus/wal
  - /mnt/prometheus/chunks_head

Prometheus appends incoming samples to the write-ahead log and keeps the newest, still-in-memory data in chunks_head, compacting both into immutable on-disk blocks roughly every two hours. Copying those two directories while Prometheus is running means reading files that are being actively appended to, so the copy is torn — and a torn WAL is worse than no WAL, because Prometheus replays it at startup.

To be exact about what this costs: the excluded data is not recovered on startup. Prometheus creates empty directories and the samples that were in them are gone, so a restore loses up to about two hours of the most recent metrics. That's an accepted trade — the completed blocks hold everything older, and a consistent restore missing two hours beats an inconsistent one that may not load at all.

Reading data that belongs to other containers

Backing up kontti means borgmatic reading application data owned by thirty-odd other containers, and that runs straight into SELinux.

Podman's :Z mount option stamps a private MCS category onto the mounted content, so only the container holding that label can read it. Twenty-five mounts on kontti use :Z — which is the correct default, and precisely why borgmatic came back with Permission denied on Plex and immich-postgres.

The obvious fix is the wrong one. Mounting appdata :ro,z — lowercase, the shared label — relabels those trees so anything can read them, and that fails in two separate directions:

• Every :Z container relabels its own tree back on its next start, so the two fight on every run. For Plex alone that is 150,000+ files being relabelled back and forth.
• A shared label lets any container read every other container's data. Solving a backup permission error by removing the isolation between all the services is a bad trade.

What's used instead is SecurityLabelDisable=true on the borgmatic container alone. That container already runs as root and can read those trees regardless, so disabling labelling for it changes nothing about what it can reach — while every other container keeps its private category. It's the narrowest change that solves the problem, rather than the one that makes the error message go away.

Off-site copy and the 3-2-1 rule

Borgmatic covers the on-site half of the strategy — the live data on each server plus the Borg repository on the NAS. The off-site half is handled by the NAS itself: Synology Hyper Backup replicates the Borg repository to Backblaze B2. That satisfies the 3-2-1 rule — three copies (live data, NAS, B2), on two distinct media, with one copy off-site — without the servers needing cloud credentials of their own.

graph LR
    src[kontti / monitoring<br/>live data] -->|Borg over SSH| nas[Synology DS920+<br/>on-site repo]
    nas -->|Hyper Backup| b2[(Backblaze B2<br/>off-site)]

Restricting the NAS key

Both servers reach the NAS with an SSH key, and for a long time that key was simply an ordinary login key on an account in the administrators group. Reviewing it made the blast radius uncomfortably clear: because Podman runs rootful on both hosts, a container escape reaches host root, and host root reaches this key — which meant a shell on the NAS, and from there the Borg repository, the media share and Hyper Backup's configuration. One key defeated all three copies, including the off-site one.

The fix is a forced command on the NAS side of the key:

command="/usr/local/bin/borg serve --restrict-to-repository /volume1/borg-backup/<host-a>.borg --restrict-to-repository /volume1/borg-backup/<host-b>.borg",restrict ssh-ed25519 AAAA...

restrict removes port forwarding, agent forwarding and PTY allocation, and the forced command means the key can only ever start borg serve — it can no longer run whatever the client asks for. The two --restrict-to-repository flags then confine that server to exactly the two repositories it is meant to write.

Verified after the change: an SSH command that used to return a shell now just prints Borg's usage text, borg info against any other path returns Repository path not allowed, and backup, prune and compact still work from both hosts. This one lives outside Ansible — it's edited through DSM on the NAS — which is exactly why it is written down here.

Restore testing

Backups are only as good as the last successful restore. The Prometheus deadman switch confirms that backups run, but proving they restore is a separate exercise.

On the Proxmox side this is automated: the restore_test.yml playbook restores the latest vzdump of a guest to a fresh, unused VMID, boots it with the network isolated, checks that it reaches running, then tears the test guest down — so it never touches the production guest.

Automated canary restores

The Borg side used to be the open gap; it is now automated with a canary file. The pre-backup hook writes a timestamp and the hostname into the backup source directory, so every archive contains a file that says exactly when it was made. A weekly borgmatic-restore-test.timer (Wednesdays at 09:00, deliberately clear of both the 04:00 backup and the Sunday integrity check, which holds a repository lock for the better part of an hour) then extracts just that one file back out of the newest archive and checks what actually landed on disk:

• borgmatic_restore_test_status — did the extract produce a readable canary at all
• borgmatic_restore_test_canary_age_seconds — how old the content is, which catches the case where the extract succeeds but the archive it came from is stale
• borgmatic_last_restore_test_timestamp_seconds — did the test itself run

All three go to Pushgateway on every run, including failures. backup.rules.yml then alerts on BackupRestoreTestFailed (extract broken, or the canary older than two days), BackupRestoreTestStale (no successful test in nine days) and BackupRestoreTestMetricsMissing — a deadman for the deadman, firing when a host that is otherwise up has pushed no restore-test metrics for ten days. Without that last one, a broken test would go quiet and the other two rules would simply never fire.

Two design details are worth calling out, because both were learned the hard way:

• The test verifies the file, not the exit code. The borgmatic container's init shuts down with SIGTERM and exits 143 on success and on failure, so gating on the exit status reported failures for restores that had worked perfectly. Checking what appeared on disk is both correct and the stronger assertion.
• It restores into its own scratch directory, created and removed by the script alone, never into live application data — a weekly automated job must not be able to touch a path a human might also be using.

This exists because "did the backup run?" is the wrong question. Two silent failures were only ever caught by inspecting archives by hand: borgmatic died in argument parsing for 45 days, and one host shipped 2-file, 4 kB archives for roughly four months. Both looked perfectly healthy to every metric that only asked whether the job had run.

Two gaps remain, and they differ in kind.

The first is the off-site leg. The B2 copy is a Hyper Backup container, which needs Hyper Backup Explorer and a large download to open, so restoring from it stays a manual exercise rather than a weekly automated one.

The second is coverage. Everything above applies to the two hosts that back up with borgmatic. The Home Assistant host runs Supervisor's own daily backup on an entirely separate path: the backups themselves happen, but none of the verification reaches them — no deadman switch, no integrity check, no restore test. That is worth stating plainly right here, because of what comes next: the incident that justifies this entire approach happened on the one host the automation still doesn't cover.

The retention half of the strategy, though, has already been validated the hard way. An NVMe failure on the Home Assistant host left the newest backup corrupt — it had been written while the drive was already degrading — and recovery depended entirely on an older, intact archive held off the failing disk. That incident is what turned "keep a history off-host, don't trust the latest file" from a principle into a verified fact — and it is why the canary test above measures the age of the restored content rather than only whether the extract succeeded.

My experience

Backups have been reliable, and Alertmanager plus the Pushgateway integration catch any run that goes missing or fails. That half has needed almost no attention.

Where I went wrong was trusting that signal further than it went. "Reliable" only ever meant the job had run — and twice it had run without doing anything useful. Borgmatic once died in argument parsing and stayed dead for 45 days, and one host quietly shipped archives of two files and a few kilobytes for months. Both looked perfectly healthy on every graph I had, because every graph I had was answering "did borgmatic run?" I found them by opening archives by hand, not by being alerted.

That is what moved restore testing off the to-do list and into a timer. The canary is deliberately tiny — one file, extracted weekly — because a drill I have to remember to run is a drill that doesn't happen. Making it check how old the restored content is, rather than just whether the extract succeeded, came straight out of the NVMe failure in July: a backup can restore perfectly and still be the wrong backup.

The off-site copy sits somewhere between tested and untested. I pulled the archive down from B2 once, a while back, and opened it — so I know the replication produces something readable at the far end, which is more than nothing. What I have never done is restore from it. Hyper Backup's format makes that a deliberate afternoon rather than something I can put on a timer, so it stays the one leg I can't claim to have proved end to end.