Backup

Both servers — monitoring and kontti — are backed up with borgmatic running in a Podman container. Backups go to the Synology DS920+ over SSH using the Borg protocol.

Schedule

Two systemd timers control when borgmatic runs:

• borgmatic.timer — triggers a backup daily at 04:00. Persistent=true ensures a missed backup runs on the next boot.
• borgmatic-check.timer — runs a full Borg integrity check every Sunday at 10:00, with a random delay of up to 10 minutes to avoid a sharp load spike.

The check runs in a separate container (borgmatic-check) so it can run independently of the backup schedule.

Pre- and post-backup hooks

Before each backup, a pre-backup script stops any services that need to be quiesced and exports specified Podman volumes to tar archives in the backup source directory. After the backup completes, the post-backup script restarts the services and cleans up the exported files.

This ensures that databases and other stateful services are in a consistent state when Borg reads the data.

Monitoring integration

Borgmatic reports its status to Prometheus via Pushgateway after each run:

# On success
echo "borgmatic_last_run_status 0" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>
echo "borgmatic_last_run_timestamp_seconds $(date +%s)" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>

# On failure
echo "borgmatic_last_run_status 1" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>

The backup.rules.yml alert rule fires if the last successful backup timestamp is too old, catching cases where the backup silently failed or the timer didn't run.

Prometheus data exclusions

When backing up the monitoring server, the Prometheus data directory is included but the WAL (write-ahead log) and chunks_head directories are excluded:

exclude_patterns:
  - /mnt/prometheus/wal
  - /mnt/prometheus/chunks_head

These directories contain in-flight data that Prometheus hasn't yet flushed to disk blocks. Copying them while Prometheus is running would produce an inconsistent snapshot — and since they're reconstructed automatically on startup, there's no value in backing them up.

Off-site copy and the 3-2-1 rule

Borgmatic covers the on-site half of the strategy — the live data on each server plus the Borg repository on the NAS. The off-site half is handled by the NAS itself: Synology Hyper Backup replicates the Borg repository to Backblaze B2. That satisfies the 3-2-1 rule — three copies (live data, NAS, B2), on two distinct media, with one copy off-site — without the servers needing cloud credentials of their own.

kontti / monitoring ──Borg over SSH──► Synology DS920+ ──Hyper Backup──► Backblaze B2
        (live data)                    (on-site repo)                  (off-site)

Restore testing

Backups are only as good as the last successful restore. The Prometheus deadman switch confirms that backups run, but proving they restore is a separate exercise.

On the Proxmox side this is automated: the restore_test.yml playbook restores the latest vzdump of a guest to a fresh, unused VMID, boots it with the network isolated, checks that it reaches running, then tears the test guest down — so it never touches the production guest.

The remaining gap is on the Borg side. End-to-end restore drills for service data — recreating a service from a Borg archive on a clean path — have not yet been performed. This is a known, deliberate gap and the highest-value next step for the backup setup.

My experience

Backups have been reliable. I monitor them through Alertmanager, which catches any missed or failed runs. The Pushgateway integration works well for this.

Restore testing is still on the to-do list — the backups run and are verified by integrity checks, but I haven't yet done a full end-to-end restore drill.