leapmotion

writing   2026-02-17   4 min   homelab, monitoring, backups

Mail only when it breaks

For a year my homelab had Prometheus, Grafana, Loki and six helper containers. I looked at the dashboards perhaps once a month, usually after something had already broken. In September I deleted all nine containers in one afternoon and did not replace them for a week. Nothing got worse. Then I put back the parts that had actually been missing.

What was missing was not graphs

The lab is a Proxmox host with eleven guests, a NAS, a Pi at a second site, and a VPS. Before the teardown, the honest state was:

  • no guest backups at all
  • one nightly “backup successful” mail from a job that backed up nothing useful
  • daily mails from a container updater, because containers updated daily
  • a dead notification endpoint that nobody had noticed for months, because it failed silently

None of that is a dashboard problem. It is a “what do I want to hear about” problem.

Rule 1: every mail is an error

Three sources send mail: the backup server, the hypervisor, and the container updater. Each one now has the same policy. The backup server’s notification matcher only fires on warning and error; the weekly prune, garbage collection and verify runs stay quiet when they succeed. The hypervisor’s built-in matcher is disabled and replaced with one that matches on severity or on the package-updates type, so I hear about failed backups and pending updates and nothing else. The container updater got a notification template that produces text only when something .Failed, and an empty message is not sent.

The nightly “successful” mail is gone. A mail in the inbox now means I have to do something.

One oddity worth knowing if you relay through your own mail account: every mail showed up twice. The server had sent exactly once, the log said so. The second copy was my provider putting the relayed message into “Sent”, and the conversation view stacking both. Not a bug anywhere, but it looks like one.

Rule 2: backups exist, and restores are tested

The nine containers had never included a backup. Proxmox Backup Server now runs in its own container and backs up all eleven guests at 02:30, with pruning after each guest, garbage collection at 05:00, and a weekly verify. Retention is seven daily, two weekly, one monthly, and the same numbers are configured on both the hypervisor side and the backup server side, because they drift otherwise.

Three things cost time:

  • Token permissions are the intersection of the user’s and the token’s. Setting the ACL on the token alone gives a clean HTTP 200 and an empty datastore list. The role goes on the user as well.
  • The NAS export uses root_squash, so every chown from the backup server fails, and it needs those for its chunk store. CIFS with a fixed uid instead of NFS solved it without touching the NAS.
  • Discard. One VM ran without discard=on. The first fstrim freed 261 GB, most of it deleted download data that would otherwise have been backed up every night as incompressible noise. A 350 GB VM now costs 57 GB on the backup store.

The VPS backs up separately with restic to the NAS over SFTP through a mesh VPN. Its script does not dump the database itself; it copies the dump that a separate job produced, and it aborts if that dump is older than 26 hours. Without that check, a silently dead dump job would back up yesterday’s file every night with a green timer. The restore was tested the day it was set up, and tested again when the transport changed, because a different path makes the restore an open question again.

Rule 3: one page that says what is up

The dashboards came back as one small thing: a lightweight monitoring hub with an agent on every host, and alerts for status, CPU, memory, disk and temperature. It mails through the same relay, under the same rule. And a start page that lists the services with a green or red dot, pulls the backup server’s task list, and shows hypervisor task errors.

That is the whole stack. Two containers, one config file each, and an agent binary per host.

What I would keep from this

Monitoring you do not read is a cost with no return. The three things I actually needed were: mail that means something, backups that exist and restore, and a single page to glance at. Everything past that was there because it was available, not because it was useful.