Background
One evening, three unrelated failures surfaced at once. My infrastructure dashboard contradicted itself — the summary counter said everything was healthy while individual cards showed services down. A Zigbee bridge had been offline for three days without a single alert. And a Fedora server had hard-locked after 73 days of uptime, taking its diagnostic evidence with it.
The easy path was three symptom fixes: recount, restart, reboot. The point of this project was refusing that path — tracing each failure to its actual root cause and pairing every fix with a layer that keeps the failure class visible forever after.
The three root causes
The dashboard was not lying — it was looking at the wrong data. The summary counter was hard-coded to one view’s group set while other views rendered their own. Worse, the down service only appeared in the views the counter ignored. The fix made the counter view-aware and added the missing endpoint to the set the summary actually watches.
The watchdog had fired — once. A Home Assistant automation had detected the Zigbee outage five minutes in, attempted a restart, and sent a push notification I missed. The restart failed, and because the automation triggered on a state change, it never fired again: the state never changed back. Three silent days. The rebuilt watchdog re-fires every fifteen minutes while the bridge is down, and a second, independent layer — Gatus with a three-failure threshold — pages through ntfy. One system recovers, a different system alerts.
The locked-up server had no warning layer armed. The hardware watchdog module was loaded but nothing was petting it. The journal flushed to disk every five minutes, so the lockup ate the last five minutes of evidence. Wake-on-LAN was set in the OS but disabled in firmware. Four changes closed the gaps: systemd hardware watchdog at 60 seconds (the machine now reboots itself), journal sync every 10 seconds (the next lockup keeps its evidence), persistent Wake-on-LAN, and TPM-backed LUKS auto-unlock so an unattended reboot actually completes — measured at 31 seconds from reboot command to responding on the network, with the disk still encrypted at rest.
A trade-off made explicitly
TPM-bound disk unlock was a deliberate choice over network-bound encryption (Tang/Clevis). The homelab threat model is drive theft — and the TPM stays with the motherboard — not whole-device theft from an off-network location. Choosing the simpler mechanism that matches the actual threat, and writing that reasoning down, mattered more to me than the theoretically stronger option.
Why it matters
Monitoring you deploy once is a feature; monitoring that survives contact with real failures is a practice. Every fix here doubles as a promise about the future: the next stuck add-on pages within minutes, the next kernel lockup reboots itself and keeps its logs, and the dashboard’s numbers mean what they say no matter which view is open. The discipline — trace first, fix second, then close the observability gap that let it happen silently — is the actual deliverable.