Backup

Both servers — monitoring and kontti — are backed up with borgmatic running in a Podman container. Backups go to the Synology NAS over SSH using the Borg protocol.

Schedule

Two systemd timers control when borgmatic runs:

• borgmatic.timer — triggers a backup daily at 07:00. Persistent=true ensures a missed backup runs on the next boot.
• borgmatic-check.timer — runs a full Borg integrity check every Sunday at 10:00, with a random delay of up to 10 minutes to avoid a sharp load spike.

The check runs in a separate container (borgmatic-check) so it can run independently of the backup schedule.

Pre- and post-backup hooks

Before each backup, a pre-backup script stops any services that need to be quiesced and exports specified Podman volumes to tar archives in the backup source directory. After the backup completes, the post-backup script restarts the services and cleans up the exported files.

This ensures that databases and other stateful services are in a consistent state when Borg reads the data.

Monitoring integration

Borgmatic reports its status to Prometheus via Pushgateway after each run:

# On success
echo "borgmatic_last_run_status 0" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>
echo "borgmatic_last_run_timestamp_seconds $(date +%s)" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>

# On failure
echo "borgmatic_last_run_status 1" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/borgmatic/instance/<host>

The backup.rules.yml alert rule fires if the last successful backup timestamp is too old, catching cases where the backup silently failed or the timer didn't run.

Prometheus data exclusions

When backing up the monitoring server, the Prometheus data directory is included but the WAL (write-ahead log) and chunks_head directories are excluded:

exclude_patterns:
  - /mnt/prometheus/wal
  - /mnt/prometheus/chunks_head

These directories contain in-flight data that Prometheus hasn't yet flushed to disk blocks. Copying them while Prometheus is running would produce an inconsistent snapshot — and since they're reconstructed automatically on startup, there's no value in backing them up.