Architecture decisions

A homelab is a series of trade-offs. This page records them — why a particular tool or pattern was chosen, and where the boundary of that choice sits. It's deliberately written as reasoning, not as a feature list.

What carries over from professional work is the method, not the conclusions. Several choices here would be wrong in a production environment and are made deliberately anyway: rootful Podman, a flat LAN, a single hypervisor node with no high availability, secrets in an Ansible vault rather than a managed secret store. They suit one person, one flat, and no availability commitment to anyone else.

Proxmox vs VMware/ESXi

I run Proxmox VE as the hypervisor rather than ESXi or Hyper-V. The deciding factors: it's open-source with no licensing friction, it has first-class ZFS and LVM-thin support, and it exposes a clean REST/pvesh API that Ansible drives directly — DNS, timezone, backup jobs and updates are all managed declaratively. For a single-node homelab the enterprise features of vSphere add cost and operational weight I'd never use.

On the licensing point specifically, it's worth being accurate rather than repeating the usual shorthand: Broadcom's post-acquisition changes ended perpetual licences in favour of subscription bundles with higher minimum core counts, which is the part that matters. The free hypervisor tier was withdrawn in February 2024 but reinstated in April 2025 with ESXi 8.0 Update 3e, so "there is no free ESXi" is no longer true. It wouldn't have changed the decision either way — a free hypervisor with no API story and no ZFS integration doesn't address what Ansible needs from this host.

The storage support isn't just a checkbox: update_proxmox.yml takes a root snapshot before a dist-upgrade and aborts on a detected major-version bump unless allow_major_upgrade: true is passed explicitly. That snapshot is worth a correction rather than a boast, because this page previously credited it to ZFS. The playbook detects the backend — a recursive zfs snapshot on ZFS roots, lvcreate --snapshot on LVM — and this host is LVM, so the ZFS path has never been the one that runs here. The version of the code that assumed otherwise didn't skip the snapshot; it failed the entire upgrade run on an unrecognised zfs error string. What survives the correction is the design principle, not the storage claim: when the requested restore point can't be taken, the run aborts loudly instead of continuing without one.

Immutable uCore vs traditional Linux

Both servers run uCore (image-based Fedora CoreOS) rather than a conventionally-managed Debian/Ubuntu host. The motivation is eliminating configuration drift: the OS is read-only, updates are atomic image swaps, and the entire machine can be rebuilt from an Ignition config plus Ansible. The cost is a less familiar operational model — you provision around the immutable base rather than editing it in place — but that constraint is exactly what keeps the servers reproducible.

The concrete cost is that you cannot simply install a package: anything the host needs either runs in a container or gets layered with rpm-ostree, which costs a reboot. That friction is doing the work — it's what makes drift visible instead of gradual.

The rebuild claim deserves a precise status rather than a confident one. There is now a written bare-metal restore runbook covering the path from Ignition through Ansible to restored data, so it is no longer only an inference from how the system is built — but it has not yet been executed end to end on real hardware. Documented, not proven. The honest ordering is that the restore testing proves the data comes back; the runbook describes the host that data would be restored onto.

Podman + Quadlets vs Docker Compose vs Kubernetes

Containers are defined as Quadlet unit files, not docker-compose.yml and not Kubernetes manifests.

vs Docker Compose — Quadlets make every container a native systemd service, so I get dependency ordering, restart policies, journald logging and timers for free, integrated with the rest of the host. No long-running Docker daemon.
vs Kubernetes — Kubernetes would be the wrong tool here. There's no multi-node scheduling, autoscaling or rolling-deploy requirement that justifies an API server, etcd and a control plane on two machines. Running k8s would mean operating the cluster instead of the services.
vs k3s and the other lightweight distributions — the obvious rebuttal to the point above, and worth answering directly, because "it's too heavy" isn't the real argument. k3s is a single binary that runs comfortably in a few hundred megabytes; resource cost was never the objection. The objection is that it is still a second control plane to operate, upgrade and back up, still cluster state that has to survive a restore, and still a manifest layer to maintain — in exchange for reconciliation and scheduling that two static hosts don't need. Quadlets also give me systemd integration (unit ordering, timers, journald, restart policy) that I would otherwise be rebuilding inside the cluster.

The deeper reason is the workloads, not the operational cost. Kubernetes exists to orchestrate things that can run as more than one instance — that assumption underpins replicas, rescheduling, horizontal autoscaling and rolling deploys. Almost nothing here can. Mealie owns a SQLite file; Miniflux and Immich each own a dedicated Postgres; Qdrant owns an on-disk vector store. A second instance pointed at the same data isn't availability, it's corruption.

A rolling deploy is the clearest case: RollingUpdate deliberately starts the new pod before retiring the old one, which for any of these means two processes writing one database. So every service would need strategy: Recreate or a single-replica StatefulSet — stop the old, start the new, accept the gap. That is precisely what a systemctl restart already does.

Hardware pins the rest down. Plex transcodes on the AMD GPU and Immich's machine learning container runs CLIP and face clustering on it; both are bound to specific device nodes on one host, as is Ollama, whose devices are passed through even though its inference currently runs on the CPU. Under Kubernetes they'd need device plugins and node affinity to keep them there — constraining away the scheduler's core function by hand.

To be fair to Kubernetes: it runs stateful singletons perfectly well, and StatefulSets, PVCs and operators exist for exactly this. The objection isn't that it can't. It's that doing it here means running a scheduler whose placement I've pinned, a replica controller I've set to 1, and a rolling deploy I've had to disable — paying for the machinery while opting out of what the machinery is for.

What I give up is worth stating plainly, since it's the first thing a design review would ask. There is no reconciliation loop: systemd restarts a unit, but nothing converges a drifted host back to its declared state the way a controller does — Ansible does that, and only when I run it. There is no scheduling, no rolling deploy, and secrets are files on disk rather than a managed store. The auto-update deadman switch is, in effect, a hand-built substitute for the part of a controller that notices reality has diverged from intent.

Rootful Podman with per-service UID hardening (not rootless)

Both servers run Podman rootful, and the isolation boundary is per-service UID pinning, Quadlet hardening and — since August 2026 — a per-container user namespace, not a rootless runtime. Rootless was considered for monitoring and deliberately rejected, which is the more interesting half of the decision.

The reason this page used to give for that rejection is gone, and replacing it quietly would be the wrong way to handle that. The old argument was that rootless as the core user bought almost nothing, because core was a passwordless sudoer — an escape would reach root in one step either way. That premise died in August 2026 when Ansible moved to a dedicated automation account and core lost passwordless sudo. Rootless is now a genuinely different blast radius rather than the same one spelled differently, so the question was re-answered from scratch instead of inherited. The answer is still rootful, for these reasons in this order:

The decisive one is observability, not security. Rootless units live in the user's systemd session, with cgroups under user.slice — and three independent alerting paths lose sight of them at once. The cAdvisor scrape derives a container's name label from a cgroup path under system.slice, and the whitelist drops anything that doesn't match, so container-down, restart and memory rules go silent. SystemdUnitFailed reads Node Exporter's systemd collector, which enumerates system units, so a failed rootless unit never appears. And whether Alloy — which reads the journal as its own unprivileged UID — can see a per-user journal at all is unmeasured. Migrating would produce about thirty instances of the failure mode this lab keeps re-learning: a rule that cannot fire looks exactly as green as one that works.
Three services can't go rootless at all, so the end state is a mixed estate rather than a rootless one. cadvisor (--privileged, mounts /, host cgroup namespace, the Podman socket), node-exporter (--pid=host, host network, / read-only) and the backup container, which reads root-owned trees across the whole estate. Mixed mode means two Quadlet locations, two container stores and two systemd layers to reason about — the exact state monitoring was once cleaned out of, orphaned rootless image storage included. That's experience here, not theory.
Rootless would require a full ownership migration; the namespace model was measured to require none. Rootless maps the invoking user to container UID 0, so every appdata tree would have to be re-derived service by service — and a Postgres that comes back with the wrong ownership simply refuses to start.
Rootless as core would re-attach every container to the identity that was just detached from root. Under a user namespace an escape lands on a subordinate UID belonging to nobody instead.
What rootless would genuinely buy, stated honestly, is one thing: Podman's own setup path (crun, conmon) would run unprivileged, so a vulnerability in container setup wouldn't start from root. User namespaces don't close that. It's a narrow delta against the four costs above.
Conditions for revisiting, written down so this isn't re-argued from taste: the cAdvisor relabel rule and whitelist extended to user.slice, a replacement for SystemdUnitFailed that covers user units, and Alloy's visibility into user journals actually measured. Until all three exist, the migration trades monitoring coverage for isolation — and here, lost coverage has consistently been noticed months late.

So instead of a rootless runtime, every service is pinned to its own unprivileged UID (Prometheus and Alertmanager as 65534, Grafana as 472, Loki as 10001, Alloy as 473), carries DropCapability=all + NoNewPrivileges=true, and — for all but Alloy — ReadOnly=true. A container escape then lands as a nobody-class UID with no capabilities and no way to gain any, rather than as core or root. cadvisor stays root deliberately, because host access is its entire purpose. The same per-UID migration is now essentially complete on kontti too, with capability hardening on top.

Removing the passwordless sudoer

The premise the rootless argument used to rest on was itself the problem, and it got fixed rather than argued around. Ansible connected as core, which Fedora CoreOS ships as a passwordless sudoer, while thirteen containers on kontti ran their PID 1 as UID 1000 — core's own UID — with no user namespace between them. A container escape was therefore one sudo away from host root.

Ansible now connects as a dedicated, password-locked, key-only ansible account with its own UID, and core has been removed from the group that grants passwordless sudo. Three details are what make it safe rather than clever:

The win is core losing passwordless sudo, not the new account existing. Creating the automation user and leaving core alone would have produced two root paths where there was one. core remains an interactive sudoer through wheel — with a password.
core's console password became load-bearing twice over, so the play refuses to remove passwordless sudo unless that account actually has a usable password set. Without the check, a locked account would be left with neither sudo nor a console — on an immutable host, that's a trip to the machine with a USB stick.
There's a chicken-and-egg interlock, and it doubles as the escape hatch. The removal is skipped when the play is running as core, because cutting sudo mid-run would fail every remaining task with sudo: a password is required. That same guard is how a host predating the change gets bootstrapped, and how the change is undone if it ever locks automation out.

The provisioning config creates the account on a fresh install, but Ignition can only add group membership, never remove it — so a newly provisioned host still carries the old exposure until the first Ansible run. That window is documented rather than papered over.

User namespaces without an ownership migration

The second half of the same change is that containers no longer share the host's UID space at all. Fifteen containers — among them Plex, Syncthing, Open WebUI, oPodSync, the Media Server MCP and Gitwatch — now carry UserNS=auto, which hands each one a distinct subordinate UID range out of a pool reserved on the host. A container that believes it is UID 1000 (or root) is some six-digit subordinate UID from the host's point of view, and no two containers share a range.

The part that made this affordable is Volume=…:idmap. An idmapped mount applies the translation at mount time, not per file, so on-disk ownership doesn't move: a container-UID-0 write still lands on the host as the same UID it always did. That has two consequences worth stating, because they're what separates this from the rootless migration rejected above — there's no data migration to perform, and removing the line is a complete rollback. Without :idmap on a writable persistent mount, subordinate ownership does reach the disk, and that's the one part a line removal can't undo.

Four inherited assumptions turned out to be wrong while migrating — about file counts, GPU passthrough, NFS and shared directories. Those, and the verification standard the migration was held to, are on the What broke, and what it took to find page.

Self-hosted vs SaaS

The boundary runs along privacy, and cost decides what's left. Photos (Immich), media, notes, and now local AI inference (Ollama) are self-hosted because the whole point is that the data never leaves the network — the plex-sync semantic search pipeline runs entirely locally for the same reason. Where a managed service touches nothing private and is free or close to it, there's no reason to run it here instead: public DNS registration, ACME certificate issuance, and this site's own hosting all sit on that side. Self-hosting is a means to control and learning, not an ideology.

Flat LAN vs VLAN segmentation

The network is currently a single flat LAN behind OPNsense, with no VLAN segmentation. For a small, trusted home environment the added complexity of inter-VLAN firewall rules and tagged switch ports isn't yet justified. The honest trade-off: a compromised IoT device sits on the same broadcast domain as everything else. VLAN segmentation — at minimum isolating IoT/ESPHome devices from servers and management — is the most defensible next step, and the point at which I'd introduce a managed switch. Documenting it as a known, deliberate gap rather than an oversight is itself part of the decision.

Segmentation below the LAN is where the effort has gone instead, because that's where it's cheap: a service that publishes no host port is only reachable from the containers sharing its Podman network. Ollama and Qdrant are the clearest case — neither can authenticate a caller, so removing their exposure is the only control that actually applies to them. The general lesson from that one: putting a login in front of a client is not a control over the backend. Open WebUI's WEBUI_AUTH secured Open WebUI and nothing else, and describing it as though it had closed the backends' exposure was the real error — a fix recorded against the wrong risk is worse than an open one, because it stops you looking.

Proxmox's own firewall stays off

pve-firewall is disabled on the hypervisor, and no playbook manages it. That's a decision, not a gap in the automation, and it's the smaller sibling of the VLAN one above: the same risk-based reasoning, applied where the cost of being wrong is much higher.

The host listens on three things — SSH (key-only, restricted to the LAN in authorized_keys), the web UI, and Node Exporter bound to a single interface. None of them is published to the internet, so the firewall would only be filtering traffic between devices that already share the LAN and could reach the host anyway. Against that, three costs:

Enabling it is not a no-op. The datacenter-level default input policy is DROP, so simply switching the firewall on breaks the Node Exporter scrape. Doing it properly means explicit accept rules for monitoring as well — more moving parts than the LAN restriction buys back.
The single VM on this host is the router. A wrong rule doesn't cost a service, it costs the entire network, including Ansible's own path back to the host. That puts firewall enablement in the same class as a reboot: something to do while physically present, not from a playbook run over SSH.
Idempotent management isn't actually available. On this Proxmox version pvesh get /cluster/firewall/options fails with a schema error, so reading the current state would mean parsing the cluster firewall file directly — hand-rolling exactly the layer Ansible is supposed to provide.

The condition for revisiting is explicit: if the host is ever exposed beyond the LAN, the firewall goes on — at the machine, and in the same session that writes accept rules for SSH, the web UI and the metrics port. Writing the condition down is the part that keeps this a decision instead of a habit.

Borg/Borgmatic vs alternatives

Backups use Borg driven by Borgmatic, targeting the Synology DS920+, which in turn replicates off-site to Backblaze B2 for a full 3-2-1 posture. Borg gives deduplicated, compressed, encrypted archives; Borgmatic wraps it in a declarative YAML config with database dump hooks and Prometheus push-gateway integration for monitoring. The deciding factor was Borgmatic's hook ergonomics: the pre- and post-hooks are what make the canary restore test and the Pushgateway deadman switch possible at all. A cluster-oriented tool like Velero would be overkill without Kubernetes.

Borg's append-only mode is deliberately not used, and it's worth being precise about that rather than claiming it as a benefit. --append-only on the server side would stop a compromised client from deleting archives — but it would also block the prune and compact actions that run as part of the daily backup, so adopting it means designing a separate retention path that runs out-of-band with different credentials. That work hasn't been done. The consequence is real and worth stating: a compromised host can still delete its own archives on the NAS. What it can no longer do — since the NAS key was restricted — is reach the media share or the off-site copy. That's defence in depth with one layer honestly missing, not a solved problem.

Restore testing used to be the other open gap on this page; it is now a weekly automated canary extract with its own deadman alert, and the remaining manual leg is the off-site Backblaze copy.

Cloudflare Workers + Git integration vs GitHub Actions

This site deploys through Cloudflare's native Git integration rather than a GitHub Actions workflow: Cloudflare watches the main branch, runs the MkDocs build in its own build environment, and publishes the static site/ output as Workers static assets (the only repo-side config is wrangler.jsonc). The alternative — a GitHub Actions job running mkdocs build and wrangler deploy — would add a workflow file, a Cloudflare API token to store as a secret, and a second system to keep in sync, in exchange for control this project doesn't need. Keeping the pipeline inside Cloudflare means one connected repository and zero CI credentials to manage.

Changes still go through a branch and a pull request rather than straight to main, even though this is a single-author site and nothing enforces it. The reason isn't review — there is no second reviewer — it's that the PR is where the diff becomes readable and where the reasoning gets written down. Since main is what Cloudflare deploys, the branch is also the only place a change exists in a state that isn't live. The cost of a bad deploy here is a quick revert rather than an outage, so this is a habit rather than a control, but it's a cheap one.

My experience

The architecture didn't emerge from a single design session — it grew from two things I already trusted. In my previous role I used OpenShift, where the operating system is atomic and every workload runs in a container, so the uCore + Podman model felt natural rather than novel. On my laptop I run Aurora Linux, part of the same Universal Blue family; the cloud-native philosophy, rolling freshness and day-to-day stability there convinced me the same approach would work for servers.

Borgmatic was an early choice, made before I had looked hard at the alternatives. It has worked reliably after the initial learning curve, so switching has never earned its cost.

Self-hosting has paid off on two fronts: cost and privacy. Keeping photos, media and AI inference on-prem means no recurring cloud bills for storage or compute, and no data leaving the network.

VLAN segmentation remains undone — honestly because this is a home lab and I live alone, the threat model doesn't yet justify the effort of tagged ports and inter-VLAN firewall rules.