Skip to content

What broke, and what it took to find

The architecture decisions page records choices made deliberately. This one is the other half of the record: the things that broke, what each one cost to track down, and what changed on the hosts afterwards.

They are collected here rather than scattered through the service pages because each one has the same shape: the unit file, podman inspect, systemd or an HTTP 200 reported healthy while the process was doing something else. What follows is what each cost to find and what changed as a result.

What the capability audit turned up

Dropping capabilities across every container on both hosts was supposed to be the easy half of the hardening work. It wasn't, because the naive rule — "the container runs as a non-root user, so dropping capabilities is free" — turned out to be wrong in both directions.

A container running as an unprivileged user already holds no capabilities in practice, which is what makes the rule sound safe. DropCapability=all changes something else: the ceiling on what that process could ever be granted. Those are two different things, and the containers below are where the difference showed.

The decisive signal is the configured user versus PID 1's real UID, read from /proc on the host rather than from the quadlet. A container with no user configured but a PID 1 running as an unprivileged UID is dropping privileges itself in its entrypoint, and needs CHOWN, DAC_OVERRIDE, FOWNER, SETGID and SETUID kept to do it — several database images fail with operation not permitted without them. Supervisor-based images need KILL on top, because the supervisor watches processes running under a different UID than its own. None of that is visible in the unit file.

Two containers needed a capability back even though they already ran as unprivileged users:

  • SYS_CHROOT for a service that drives a headless browser. The browser sandboxes its content processes in a user namespace of its own, and building that sandbox needs SYS_CHROOT — which the container can only use if the ceiling above it still allows it. Bisected with throwaway runs: unhardened fine, NoNewPrivileges alone fine, dropping all capabilities broke it, adding back SYS_CHROOT fixed it. The two settings are kept together because they cover different routes: NoNewPrivileges guards against a privileged binary arriving in a future image, while dropping capabilities lowers the ceiling itself.
  • NET_BIND_SERVICE for two images that run their own DNS resolver on port 53 inside the container. Without it the resolver crash-loops and name resolution dies inside the container while everything outside still looks healthy.

Both failures would have sailed through a shallow HTTP check — one service's API kept returning 200 the entire time its browser was broken, and the other's web UI answered normally with DNS dead. Hardened containers are therefore checked here by exercising what they actually do, not by whether the port answers — and against /proc of the running process, because a stale generated unit will happily report the hardening you asked for while the process runs without it.

Trusting the process over the management layer

That last point turned out to be the more portable lesson, because a second incident produced it from the opposite direction. Together they set the rule these two produced here: when the management layer and the process disagree, the process is right.

The first case is the stale generated unit above. The quadlet contained DropCapability, the generated unit contained --cap-drop all, systemctl is-active said running, and podman inspect reported EffectiveCaps=[] — while the running process had never picked the change up. Four signals agreed, and all four were reading configuration rather than the process. podman inspect is the subtle one: its answer was perfectly true and entirely uninformative, because it reports the same thing for any unprivileged container whether the hardening applied or not.

A container that could no longer fork

The second case cost considerably more to find. A browser-driving service went down and surfaced only as a failed probe. Everything above the process looked healthy: systemd reported active (running) throughout, Restart=always meant SystemdUnitFailed never tripped, memory was 701 MB of 31 GB, /dev/shm was empty, the disk was fine and there were no OOM messages anywhere.

The container's PID 1 was the application's own launcher, which never calls wait(). The headless browser spawns helper subprocesses per session; when those exit they reparent to PID 1, and with nothing reaping them they stay zombies indefinitely.

A zombie uses no memory, which is why nothing about the container looked short of resources — but it still occupies a slot against the process limit, and that slot is only released once something reaps it. After two days the container was at 2031 processes against Podman's default limit of 2048, with only 127 of them alive. It could no longer start a new one, so the browser failed to launch and the endpoint started returning 500.

The fix is RunInit=true, which puts catatonit in as PID 1 to reap orphans. A restart alone only resets the counter — it buys another two days.

Three things the incident pinned down:

  • systemd's process limit is not the one that bit. The unit reports a comfortable TasksMax of 35650 and looks entirely healthy; the limit that actually applied was Podman's own, one level below it.
  • Check PID 1 before adopting any image that drives a browser or other multi-process children — podman exec <container> cat /proc/1/comm. Anything that isn't an init (catatonit, tini, s6-svscan) needs RunInit=true. The unambiguous live symptom is podman exec itself answering Cannot fork.

The common thread is that every signal reporting "healthy" was describing configuration or supervision state, not work performed. systemd knows whether it started something and whether it exited; it does not know whether the thing is doing its job. That gap is exactly what ContainerRestarting and the probe rules exist to cover, and why the capability work above was verified against /proc/<pid>/status rather than against podman inspect.

What the user-namespace migration corrected

Moving fifteen containers onto their own user namespaces was mostly uneventful, which is the point — but four inherited assumptions turned out to be wrong, in both directions.

  • The config tree's size was never a reason to exclude a service. Plex was held back on the grounds that remapping ~64 GB and 150k files would be too expensive. It costs nothing: an idmapped mount is set up once when the mount is created, not per file. The real reason to be careful with Plex was GPU passthrough, and that was settled by testing rather than by argument: the render node is world-readable and a real VA-API encode succeeds inside the namespace. Network=host turned out to be no obstacle either — network and user namespaces are independent.
  • The :idmap decision is per mount, not per container. Local appdata gets it; the NFS share must not — idmapping fails there, and isn't needed, because the export squashes ownership anyway. The one behaviour that genuinely changes is that chown on the share is refused inside the namespace, so each application had to be checked for whether it relies on it. A control run showed chown already failed there without a namespace, which is the sort of thing worth measuring before blaming a change for it.
  • Two containers sharing a directory can be migrated, but only together. They get different ranges, yet both see the shared tree as the same owner because the map is an identity. The near-miss there was SSH: OpenSSH refuses a private key not owned by the running UID, so without :idmap on the key mount, pushes would have stopped silently with no alert.
  • The s6-based images are where this pays off most. Those run PID 1 as real host root by design and can't be configured out of it, so a user namespace is the only thing that moves them off host UID 0 — which it did, each to its own range, with identical behaviour before and after.

The verification standard was the same one the capability audit set: read PID 1's real UID from /proc on the host, and exercise what the service actually does — a real transcode, a real hardlink, a real push — rather than trusting the unit file or a 200 from a health endpoint.

My experience