This is part 7 of a series on building a homelab Kubernetes cluster. Part 6: Closing the Loop from Commit to Production covers the automation this database now sits alongside.

Nothing forced this one. No app needed it yet, no feature was blocked on it. Just wanted somewhere to put state for whatever gets built next, small and unglamorous on purpose, "nothing heavy." The interesting part turned out to be the question underneath the question: not whether the hardware could run a database, but what "reliable" actually means in a cluster that had never had to answer that before.

The wrong question first

The instinct was to ask where a database would perform best, in Kubernetes on the cluster's own fast local disk, or on the NAS, physically closer to actual storage. Reasonable question, wrong one to start with. What actually mattered, once said out loud: the cluster itself should be treated as disposable. Wipeable, rebuildable, no grief about starting over. The data behind anything that keeps state can't be.

That reframes the comparison completely, because it exposes something true about every stateful thing already running in this cluster: Prometheus, Alertmanager, Grafana, all of it sits on mf1's own local NVMe through a static hostPath PV, one disk, no redundancy of any kind. Fast, yes. If that disk dies, everything on it is just gone, and no amount of raw NVMe throughput changes that. The NAS, four bays deep in a redundant array, is the only thing in this entire setup that actually survives a single-disk failure. That's not a performance argument. It's the only argument that matters once "the cluster is allowed to disappear" is a real constraint and not a hypothetical.

What got built

Postgres runs as the active instance in Kubernetes, a single-replica StatefulSet pinned to mf1, the same hostPath pattern as everything else here, reachable through a LoadBalancer Service so anything outside the cluster's own network, the NAS included, has a real address to reach it at. A dedicated role handles replication, REPLICATION privilege only, nothing close to superuser, because the thing streaming a copy of this data off-cluster should never be able to do anything else with it.

Two real bugs showed up getting the primary running at all, both small, both the kind that only show up by actually trying it rather than reading the manifest twice. A local PersistentVolume, unlike a plain hostPath volume, expects its target directory to already exist on the node, it won't create one, and the pod sat in ContainerCreating refusing to explain why until describe finally surfaced a mount failure buried past everything else. Fixed by creating the directory by hand before anything tried to claim it. The second was sharper: the container came up, then immediately refused to start, could not open file "pg_hba.conf": Permission denied, because a ConfigMap mounted at 0640 isn't readable by a non-root container user unless the pod's fsGroup says so explicitly. Postgres was fine. The permissions model underneath it wasn't, and every mounted volume, real config file or plain scratch space, needed fsGroup to actually be usable by the user running the process.

A database with nothing running against it isn't tested

Getting Postgres to report ready and accept connections proves approximately nothing. It proves a container started. Whether the actual thing, a role that can connect, a schema that gets created, a query that returns the right row, works end to end is a completely different question, and the honest way to answer it wasn't to write a synthetic test against an empty table. It was to build something that would actually depend on it for real.

That became pagewatch, deliberately small: give it a URL and how often to check it, it fetches that URL on schedule, and if the content changes from what it saw last time, it emails a diff. Not a toy built to justify the database, a genuinely useful thing, the kind of small utility that's easy to want and annoying to hand-roll each time it comes up. It needed exactly the shape of state a database is for: URLs, schedules, a content baseline to diff against, a running log of what changed and when. First real workload against the new instance, and the only way to know Postgres actually worked was to watch this app use it for something.

Proof mattered more than a green checkmark here too. Added a real URL, let it establish a baseline against a real fetch. Then, instead of waiting on an external site to actually change (slow, and not something to depend on for a test), rewrote the stored baseline directly to something deliberately stale and let the next scheduled check run against it. It caught the difference, logged it, and the diff email arrived exactly as designed, colored red and green lines in the same visual language the rest of this cluster's notifications already use. Small test, but a real one: the database held state, an app changed its mind based on a query result, and a notification went out the same path Alertmanager's already been using.

The standby, proven the same way

The first draft of this post stopped here with an honest admission: the NAS-side standby was designed in enough detail to build without re-deciding anything, which role authenticates, what the bootstrap looks like, how a promotion would work, but none of it was actually running. A plan for durability isn't durability, the same lesson the last section was already about, so it seemed dishonest to round that up to done just because the runbook read well.

It's built now, and proven the same way the primary was: not by trusting that it should work, but by checking. pg_basebackup -R seeded the NAS-side data directory and wrote its own standby.signal in the same step, and the container's own log shows the exact sequence that means it actually worked, entering standby mode, consistent recovery state reached, then started streaming WAL from primary. That's the standby's side of the claim. The primary's side had to agree independently for it to count: querying pg_stat_replication on the Kubernetes instance shows a row for the NAS's address with state = streaming, not because the standby says so, but because the primary can see it receiving WAL in real time. Two independent sources, same answer, which is what "proven" actually requires here rather than just one side reporting success.

What this adds up to

A database that exists is not the same claim as a database that works, and a plan to make something durable is not the same claim as durability. Both gaps look identical from the outside, a manifest applied, a paragraph written, right up until something real tries to depend on either one. The primary cleared that bar first, an app actually needed it and got correct answers back. The standby has now cleared it too, checked from both ends instead of assumed from one.

One gap is still honestly open, and worth naming rather than letting the rest of this post imply everything's covered: a streaming standby protects against mf1's disk dying or the cluster getting wiped, but it replicates a bad DROP TABLE or an application bug just as faithfully as it replicates everything else. That needs a second, independent mechanism, a point-in-time snapshot the standby can't inherit the same mistake from, and it isn't built yet. Same rule applies to it that applied to everything else in this post: it isn't done until something proves it, not before.