uCore

Both homelab servers — monitoring and kontti — run uCore, an immutable, image-based Linux built on Fedora CoreOS. The OS is never modified in place: updates are applied as atomic image swaps and take effect after a reboot.

Why immutable Linux

With a traditional Linux server, configuration drift accumulates over time — packages installed manually, files edited by hand, state that exists nowhere except on the running machine. uCore flips this: the base OS is read-only, everything is provisioned declaratively, and the server is designed to be rebuilt from scratch using the same Ignition config and Ansible playbooks — a path now written up as a runbook, though not yet exercised on real hardware.

Initial provisioning

A new server is provisioned in two steps:

1. Generate the Ignition config from Butane

podman run --interactive --rm quay.io/coreos/butane:release \
  --pretty --strict < test.butane > test.ign

Butane is a human-friendly YAML format that compiles down to Ignition JSON. The config declares the initial user, SSH keys, hostname, keyboard layout, and the auto-rebase systemd services.

Note what the config doesn't contain: a password_hash for the core user. It's a natural thing to put in a Butane file, and the reason not to is that Ignition runs only on the first boot — so the hash sits in version control permanently while the password it sets stays live on the machine forever. First access comes from the SSH key above; the console password is set afterwards with sudo passwd core and kept in a password manager. It's worth setting, because it's the only interactive recovery path if SSH ever breaks — after a firewall change, for instance.

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 <public-key>
storage:
  directories:
    - path: /etc/ucore-autorebase
      mode: 0754
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: monitoring
    - path: /etc/vconsole.conf
      mode: 0644
      contents:
        inline: KEYMAP=fi
systemd:
  units:
    - name: ucore-unsigned-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to unsigned OCI and reboot
        ConditionPathExists=!/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/signed
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver \
          ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/unverified
        ExecStart=/usr/bin/systemctl disable ucore-unsigned-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target
    - name: ucore-signed-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to signed OCI and reboot
        ConditionPathExists=/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/signed
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver \
          ostree-image-signed:docker://ghcr.io/ublue-os/ucore:stable
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/signed
        ExecStart=/usr/bin/systemctl disable ucore-signed-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target

The two autorebase services handle the uCore bootstrap automatically. On first boot, ucore-unsigned-autorebase rebases to the unverified image and reboots. On the second boot, ucore-signed-autorebase takes over and rebases to the signed stable image. The ConditionPathExists guards ensure each service runs exactly once:

graph TD
    iso["coreos-installer<br/>+ Ignition config"] --> b1

    subgraph b1["Boot 1 — Fedora CoreOS"]
        ign["Ignition applies config<br/>user, hostname, units"] --> u1["ucore-unsigned-autorebase<br/>guard: neither marker exists"]
        u1 --> m1["touch .../unverified<br/>disable self, reboot"]
    end

    m1 --> b2

    subgraph b2["Boot 2 — uCore, unverified"]
        u2["ucore-signed-autorebase<br/>guard: unverified set, signed not"] --> m2["touch .../signed<br/>disable self, reboot"]
    end

    m2 --> b3["Boot 3 — uCore, signed<br/>ready for Ansible"]

Two properties of that flow are the whole point. Ignition runs only on the very first boot, so anything it writes is permanent from that moment — which is why no password hash goes in it. And the marker files, not the systemd unit state, are what make each rebase happen exactly once: a unit that re-enabled itself or a machine reimaged from the same config lands in the same place, because the guard reads the filesystem rather than trusting that the disable step ran.

2. Boot from ISO and install

# Write ISO to USB
sudo dd if=fedora-coreos-*.iso of=/dev/sdX bs=4M status=progress && sync

# Serve Ignition file over HTTP and install
python3 -m http.server
sudo coreos-installer install /dev/nvme0n1 \
  --insecure-ignition \
  --ignition-url http://<YOUR_IP>:8000/test.ign

On first boot, two systemd services handle the uCore rebase automatically — first to the unsigned image, then to the signed stable image — rebooting between each step. After that the server is ready for Ansible.

Ansible

All ongoing configuration is managed with Ansible from the ucore-ansible repo. It connects as a dedicated, key-only ansible account rather than as the interactive core user — see Removing the passwordless sudoer for why that matters more than it looks. The playbook is run from the ansible/ directory:

# Both servers
ansible-playbook install.yml

# Single server
ansible-playbook install.yml --limit kontti

# Single role
ansible-playbook install.yml --limit kontti --tags media

# Only config files, no service restarts
ansible-playbook install.yml --tags config

Roles map to tags, making it easy to apply targeted changes without running the full playbook.

Podman and Quadlets

All services run as Podman containers. Instead of docker-compose, containers are defined as Quadlet files — systemd unit files that Podman generates into actual services at boot. Quadlet files live under /etc/containers/systemd/.

Both servers run Podman rootful. Isolation doesn't come from a rootless runtime — it comes from pinning each container to its own unprivileged UID:

• monitoring runs each service under a dedicated non-root UID (Prometheus and Alertmanager as 65534, Grafana as 472, Loki as 10001, Alloy as 473), with capabilities dropped and the root filesystem read-only where the service allows it. Rootless was rejected on purpose, and re-argued from scratch once core lost its passwordless sudo: rootless units live under user.slice, where three separate alerting paths stop seeing them, and cadvisor and node-exporter need host access and cannot go rootless at all. The full reasoning is on the Architecture decisions page.
• kontti historically ran everything as root:root. That migration is now essentially complete: its services were moved one at a time to unprivileged UIDs — most to 1000 — and as of August 2026 every container except two also carries NoNewPrivileges=true and a dropped capability set. Its GPU-bound and host-path workloads (the AMD GPU for media transcoding and photo machine learning) are the reason it started rootful, but that is a device-access decision, not a rootless-vs-root one.

Since August 2026 the UID pinning also sits inside a per-container user namespace. Fourteen containers carry UserNS=auto, so each one gets its own subordinate UID range: what a container sees as UID 1000 — or as root — is a six-digit UID belonging to nobody on the host, and no two containers share a range. Persistent local mounts are declared :idmap, which applies the translation once at mount time rather than per file, so nothing on disk changes ownership and removing the line is a complete rollback. The Architecture decisions page covers what that closed and what it cost to verify.

cadvisor stays root on both hosts, deliberately — it is --privileged with the host cgroup namespace and a mount of /, so direct host access is its whole job. The one other exception on kontti is a pod infrastructure container, where a per-container user simply doesn't apply.

node-exporter used to be listed alongside it as "needs root". Measuring it disproved that: on kontti it already ran as the image's own nobody user (65534) with an empty effective capability set, because everything it reads through --pid=host and its read-only mount of / is world-readable anyway. It has since been hardened like the rest, with its metrics verified still flowing. Checking what the process actually runs as, before assuming it needs privilege, is what the capability audit records in full.

/etc/containers/systemd/
  plex.container
  immich-server.container
  immich-postgres.container
  ...

This means every container is a native systemd service: systemctl start plex, journalctl -u plex, restart policies, dependencies — all handled by systemd.

Automatic updates

podman-auto-update runs on a daily timer and pulls new image versions for any container with AutoUpdate=registry set. The timers are staggered per host — kontti at 05:30, monitoring at 08:00 — so the host that collects the metrics isn't restarting its own containers at the same moment the other host is updating.

After an update, a script sends a Telegram notification listing the containers that changed. That notification is not monitoring, though, and treating it as such was the flaw: it only fires when something did update, so a night with no updates and a night where the timer never ran look identical from the phone's point of view.

A second script therefore pushes the run's outcome to Pushgateway on every execution:

• podman_auto_update_status — whether the unit succeeded
• podman_auto_update_containers_updated — how many images changed (0 is completely normal and is deliberately not alerted on; it exists so a later breakage can be correlated with a night something updated)
• podman_auto_update_last_run_timestamp_seconds — proof the timer fired at all

It runs from ExecStopPost rather than ExecStartPost, because ExecStartPost only runs when ExecStart succeeded — a failed auto-update would push nothing and look exactly like a timer that never ran. ExecStopPost always runs, and it has systemd's $SERVICE_RESULT available to distinguish the two. podman.rules.yml then alerts on a failed run, a stale timestamp, or the metrics disappearing entirely.

How far the images are pinned

Auto-updating from :latest means accepting whatever the next tag resolves to, including a major version. Every image in the fleet was therefore checked against its registry, resolving the candidate tags rather than assuming which ones existed — and the assumption going in turned out to be wrong. Most of these images publish no floating major or minor tag at all: only channel tags and exact patch versions. Pinning those would freeze them permanently and cut off security updates, so they stay on :latest with a comment recording why.

Three images had a genuine floating major and were pinned to it — Node Exporter, Pushgateway and Homepage, each verified to resolve to the same digest as :latest, so the pin changed nothing on the day. Pushgateway matters most of the three: it carries the backup deadman switch, so an unattended major jump would take out precisely the signal that says backups are alive.

Two judgement calls are worth recording, because both cut against the obvious rule:

• A tag existing doesn't mean it's moving. One image had :1, :2 and :2.0 available, but comparing manifest creation dates showed :latest was months newer than any of them — they were frozen, not floating. Pinning would have been a six-month downgrade dressed up as a safety measure.
• On 0.x software, a minor pin fails silently. A :0.60 pin stops receiving updates the moment 0.61 appears, and nothing reports it — one image had already sat seven months behind unnoticed. :latest risks a breaking update, which shows up loudly in ExporterDown and ContainerRestarting. Given a choice between a loud failure mode and a silent one, take the loud one.

Scheduled reboots

Since uCore applies OS updates as image swaps, a reboot is needed to activate them. A systemd timer triggers a scheduled reboot so updates are picked up automatically without manual intervention. The cadence is set per host to match each server's role:

monitoring reboots daily to stay current, while kontti reboots only once a week to avoid interrupting long-running media tasks and active streams more often than necessary.

My experience

I chose an atomic OS for the servers because the same model had already proven itself on my laptop, which runs Aurora Linux — part of the same Universal Blue family as uCore. It also mirrors what I work with professionally: OpenShift uses the same pattern — an immutable base OS with everything running in containers — so the mental model carried straight over.

The genuinely new thing was Ignition; first-boot provisioning was a concept I hadn't worked with before, and getting the Butane config right took some iteration. Running Borgmatic in a container also turned out to be more awkward than running it natively, since it wants host paths and scheduling that are simpler to wire up outside a container. Once the server was up, though, deploying the applications with Ansible was smooth.

One thing to watch for: when you remove or change a service, it's worth checking that no old Quadlet units, containers, or images are left dangling — removing the Ansible task doesn't clean those up for you.

I haven't actually had to reinstall a server from scratch yet, so the "rebuild from nothing" capability is still theory for me — but it's the design I trust, and the Ignition-plus-Ansible path is there when I need it. Day to day, the servers have been genuinely appliance-like: as long as they have power, the OS and the applications keep themselves up to date.