Monitoring

The monitoring stack runs on the dedicated monitoring server — a repurposed Lenovo laptop running headless with the lid closed. Sleep is disabled via systemd-logind so it stays on regardless of lid state:

[Login]
HandleLidSwitch=ignore

This is deployed by Ansible when disable_sleep: true is set in the host's variables. It covers metrics, logs, alerting, and endpoint availability across the entire homelab.

All services run as Podman containers on a shared internal network, provisioned with Ansible. Containers are defined as Quadlets under /etc/containers/systemd/. Podman runs rootful, but each service is pinned to its own unprivileged UID (Prometheus and Alertmanager as 65534, Grafana as 472, Loki as 10001, Alloy as 473) with capabilities dropped and a read-only root filesystem where possible — so a container escape lands as a nobody-class UID, not as root. cadvisor stays root, because --privileged host-level access is its entire purpose. node-exporter was assumed to need root for the same reason until it was actually measured; what that turned up is described on the uCore page. Why all of this was chosen over a rootless runtime is explained under Architecture decisions.

The full stack

Two collection paths — metrics and logs — converge on the monitoring server, where Grafana reads both and Alertmanager fans out anything that breaches a rule:

graph LR
    subgraph sources[Sources across the homelab]
        ne[Node Exporter<br/>hosts + OPNsense]
        cad[cAdvisor<br/>monitoring + kontti]
        pve[PVE Exporter]
        snmp[SNMP Exporter<br/>Synology DS920+]
        bb[Blackbox Exporter<br/>HTTP / ICMP / TCP]
        pg[Pushgateway<br/>backup jobs]
        alloy[Grafana Alloy<br/>monitoring, kontti, HA]
    end

    subgraph mon[monitoring server]
        prom[Prometheus]
        loki[Loki]
        graf[Grafana]
        am[Alertmanager]
    end

    ne & cad & pve & snmp & bb & pg --> prom
    alloy -->|logs| loki
    alloy -->|metrics| prom
    prom --> graf
    loki --> graf
    prom -->|alert rules| am
    am -->|Telegram| tg[Phone]

Metrics

Prometheus scrapes metrics every 15 seconds from targets across the homelab:

blackbox_ai is a second, deliberately minimal Blackbox Exporter running on kontti. When Ollama and Qdrant were taken off the LAN, the probes running from this host lost their route to them and began alerting on healthy services — no relabelling rule can cross a network boundary. Running a probe inside that network restores the coverage without republishing either backend; it exposes only its own probe_* metrics. Dropping the probes instead would have left both services covered by container-level rules alone, and "container up, application not serving" is exactly what those rules cannot see.

Designing the probes

A blackbox probe is a request you make of your own service on every scrape, and it is easy to forget that this is real load.

One service's /health endpoint launched a full headless browser per request and fetched an external page through it. At the global 15-second scrape interval that worked out to roughly 5760 browser launches a day from monitoring alone — about 20× the service's actual traffic, and ultimately what filled its PID table. It now has its own blackbox_slow job at a two-minute interval.

Two minutes is deliberate rather than round: five would sit exactly on Prometheus's five-minute staleness window, while two still gives a for: 5m rule three samples to work with, putting detection at around seven minutes.

The probed endpoint changed too, because /health was not a check that service could reliably pass — it hardcoded an external URL and waited for the browser to reach networkidle, which that page never quite does. Over six hours, 13 of 62 samples failed, about 21%, on a service that was working perfectly throughout. The probe now exercises the real endpoint against a page with no subresources: six runs out of six between 1.50 and 1.63 seconds. Two regular expressions assert both the wrapper's status field and the fetched page's status code, so a browser that returns an error page still fails the probe rather than passing it.

Three diagnostics from that work generalise:

• Blackbox uses the smaller of its module timeout and Prometheus's scrape_timeout. This silently defeated the first attempt at the fix: the job carried a 30-second scrape_timeout, but the module's own five-second timeout won, so every "failure" was blackbox cutting the probe off early. Keep the job's timeout just above the module's, and change them together.
• Read probe_duration_seconds before blaming the service. Every failed sample was exactly 5.00 s while successes sat at 2.9–3.1 s. A constant duration on a round number is a timeout; a genuinely broken backend produces scattered durations or a non-2xx status.
• Moving a target between scrape jobs breaks every rule keyed to the old job label — silently. After the move, a query against the old job kept returning data for about five minutes. That is staleness residue, not configuration. Confirm against /api/v1/targets, which lists only live targets, and prove the new expression really selects the target by inverting it: == 1 should return it, == 0 should return nothing. Same principle as the deadman rules — an expression that matches nothing looks exactly as green as one that works.

Alert rules

Prometheus evaluates alert rules every 15 seconds. Rules are organized by category:

• node.rules.yml — CPU, memory and disk thresholds, plus NodeDown and ExporterDown
• container.rules.yml — container-down and crash-loop detection
• systemd.rules.yml — failed systemd units (SystemdUnitFailed)
• probe.rules.yml — endpoint availability failures, plus SSLCertExpiringSoon, which warns 14 days ahead and reads the expiry from the certificate served on the wire rather than from a file on disk — so it catches a renewal that succeeded but never got loaded
• proxmox.rules.yml — Proxmox VM and storage alerts
• backup.rules.yml — missed or failed backups, and restore-test results, via Pushgateway
• synology.rules.yml — NAS health and disk status
• logging.rules.yml — log shipping stopped or entries dropped
• podman.rules.yml — deadman switch for podman-auto-update
• oneshot.rules.yml — deadman switch for scheduled one-shot jobs

An excerpt from node.rules.yml shows the shape of a rule — the thresholds, the for duration that debounces flapping, and the templated annotations that land in the Telegram message:

groups:
  - name: node
    rules:
      - alert: NodeDown
        expr: up{job=~"node|opnsense|proxmox|kontti"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node unreachable"
          description: "Node Exporter on {{ $labels.instance }} has been unreachable for 2 minutes."

      - alert: DiskSpaceLow
        expr: >
          (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint=~"/|/sysroot|/var"}
          / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint=~"/|/sysroot|/var"}) * 100 < 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk space low"
          description: "{{ $labels.instance }} mountpoint {{ $labels.mountpoint }} has {{ $value | printf \"%.1f\" }}% free."

      # NAS uses absolute thresholds — on a 47 TB volume, percentages are too coarse
      - alert: NasDiskSpaceLow
        expr: node_filesystem_avail_bytes{mountpoint="/var/mnt/nfs-data"} < 2199023255552
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "NAS disk space low"
          description: "{{ $labels.instance }} NAS has {{ $value | humanize1024 }}B free (under 2 TB)."

      - alert: HighCPULoad
        expr: >
          (1 - avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) * 100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU load"
          description: "{{ $labels.instance }} CPU usage is {{ $value | printf \"%.1f\" }}% (over 85% for 10 min)."

Closing two blind spots

Two rules exist specifically because a real outage slipped through the gaps the obvious alerts leave — the kind of "why" this site tries to capture rather than just listing what's configured:

• ExporterDown (up{job=~"pve|snmp|cadvisor|pushgateway|blackbox|blackbox_ai|alertmanager|alloy"} == 0) — an exporter being down can't be caught by a rule written over the metrics it produces, because the metric is then absent rather than zero. The Proxmox VE exporter was down for three and a half days this way after an auto-update changed its image user and left its config file unreadable; ProxmoxNodeDown (pve_up == 0) never fired because pve_up had simply vanished from the time series. ExporterDown works because Prometheus synthesises up for every scrape target whether or not the target answers — so the signal doesn't depend on the thing being monitored still functioning. Every exporter added since is added to this list too, which is why blackbox_ai appears in it.
• ContainerRestarting (max by (instance, name) (changes(container_start_time_seconds{name!=""}[15m])) > 5) — a crash-looping container is invisible to ContainerDown (its container_last_seen keeps advancing as the container is recreated every few seconds) and to SystemdUnitFailed (Restart=always never lets the unit settle into failed for the 15 minutes that rule requires). This rule counts restarts instead. The threshold isn't a guess: checked against a real crash-loop day over a three-hour window, it picked out exactly three containers — one at 364 restarts, one at 138, one at 9 — while ordinary deploy restarts stayed at one to three. Five sits in the empty gap between the two populations.

SystemdUnitFailed itself is built on Node Exporter's systemd collector rather than on container metrics, so it catches stopped or removed services that the container-level rules never see.

When the monitoring itself creates the fault

ContainerDown and ContainerRestarting are both aggregated with max by (instance, name) rather than evaluated per series, and the reason is a false positive worth recording.

cAdvisor reports several cgroups per container. The relabelling that derives a name label from the cgroup path once matched all of them — including systemd's transient /.control cgroup, which appears during a restart and then disappears. Because it disappears, its container_last_seen freezes, and a per-series rule reads that frozen timestamp as a container that has stopped being seen. On 2026-08-01 an ordinary --tags service deploy produced two immediate critical alerts, for pve-exporter and snmp-exporter, while podman ps showed both as Up (healthy).

The relabel regex has since been tightened so only the base /system.slice/<unit>.service cgroup gets a name, which removes the cause. The aggregation was deliberately kept anyway: it costs nothing, it covers the case where systemd or Podman introduces some new sub-cgroup later, and it cannot mask a genuine failure — if a container really dies, all of its series go stale, so the maximum goes stale with them.

The general point is one worth carrying: an alert firing on a healthy system is a defect in the same way a missed outage is. Both teach you that the rule was measuring something other than what you meant.

Logs

Grafana Alloy runs on monitoring, kontti, and the Home Assistant host, collecting logs from each host's systemd journal and forwarding them to Loki on the monitoring server. The agent on kontti is deployed by the kontti_monitoring role alongside cAdvisor and Node Exporter, so the application server's logs and container metrics make it into Grafana even though the storage backends live elsewhere. The Home Assistant host runs its own Alloy agent that ships both its journal logs to Loki and system metrics to Prometheus via remote write, filtered down to the Home Assistant unit. Noisy but harmless log lines are filtered out before reaching Loki to keep the log volume manageable, and logs are tagged with host, systemd unit, and container name for easy filtering.

The log-shipping blind spot

Log collection has its own version of the gap described above, and logging.rules.yml exists to close it. An Alloy agent that is running but shipping nothing is invisible: ContainerDown and ContainerRestarting catch the container dying or crash-looping, and ExporterDown catches the process disappearing, but "up and doing nothing" matches none of them. The logs would simply stop arriving in Grafana — and because an empty log view looks exactly like a quiet system, that goes unnoticed until the logs are actually needed, which is precisely during an incident.

• LogShippingStopped (increase(loki_write_sent_entries_total[1h]) == 0) — the window is an hour rather than thirty minutes so a genuinely quiet night can't trip it; combined with for: 30m, logs have to be completely stalled for an hour and a half before it fires.
• LogEntriesDropped (increase(loki_write_dropped_entries_total[1h]) > 0) — a different failure mode entirely. Rate limiting, an over-long line or a full queue drop lines without stopping the flow, so the first rule never sees them, but log data is being lost all the same.

Unlike the backup and auto-update deadman switches, these need no absence guard of their own: they're scraped metrics rather than pushed ones, so a metric that vanishes means Alloy isn't answering — which is ExporterDown's job.

Alertmanager

Alertmanager receives alerts from Prometheus and handles deduplication, grouping, and routing. Notifications are sent to Telegram.

The metrics-to-notification path looks like this:

graph LR
    ne[Node Exporter] --> prom[Prometheus]
    cad[cAdvisor] --> prom
    bb[Blackbox Exporter] --> prom
    prom --> graf[Grafana]
    prom -->|alert rules| am[Alertmanager]
    am -->|Telegram| tg[Phone]

Alert policy

There's nobody else to notify, so the routing is tuned to stay useful without being noisy. Alerts group by alertname, wait 30s to batch related firings, and repeat at most once an hour while a condition persists. Critically, a yö (night) mute interval silences Telegram notifications between 20:00 and 07:00 — alerts still fire and resolve in Alertmanager, but they don't wake me for anything short of an outage I'd notice anyway. How quickly an alert actually gets looked at depends on what it is and what time there is for it — the severity split exists to inform that judgement, not to promise a response time.

Grafana dashboards

Grafana runs the Enterprise edition image (used here without a paid licence, so it behaves as open-source Grafana). It is provisioned with a fixed set of dashboards so the visualisations are reproducible from Ansible rather than clicked together by hand. Community dashboards are downloaded at a pinned revision and patched (datasource, mountpoint, variables) to fit this environment; the rest are maintained locally as JSON.

My experience

I previously used Zabbix for monitoring and wanted to move to more modern tooling. Prometheus, Alertmanager and Loki are common in the Kubernetes world, and Loki is significantly lighter than the Elastic stack. I used AI to help set up the stack.

Alerts have been useful once I got them tuned to my liking — the initial configuration took some iteration. The Grafana dashboards were more of a learning project; I rarely look at them in practice because the environment is stable enough that troubleshooting is infrequent. Log collection has been similar — mostly a learning exercise so far, but it's convenient to have metrics and logs in the same system if I ever need to dig into an issue.

It's worth being honest about how most of the gaps on this page were found: not by the monitoring itself. The exporter that had been down for three and a half days, the deadman rule that could never fire, the services that turned out to have no probe at all — those came out of review sessions where I go through the configuration with an LLM and ask what doesn't hold up. The monitoring catches what it was built to catch. A review catches what I didn't think to build a rule for, and those are different problems.