This is part 1 of a series on building a homelab Kubernetes cluster. Part 2: A GPU Node That Comes and Goes covers adding a second, GPU-capable node. Part 3: Real DNS Names, Real Certificates covers moving off a bad domain and getting real Let's Encrypt certs for internal-only services.

The machine I called "the Linux mini-PC" in Part 4 of the OPNsense series is now mf1, node one of a Kubernetes cluster. It used to run Vector, Loki, Prometheus, OpenSearch, and Grafana off a docker-compose file. Same box: AMD Ryzen 9 6900HX, 30GB RAM, 900GB+ disk. Different job now.

This is the writeup of turning it into a cluster, the requirements I set before touching anything, and the two things that actually fought back: LoadBalancer IPs and DNS.

Requirements, decided before any commands ran

1. Start with one box, but don't paint myself into a corner. Whatever I set up needed to be a real multi-node-capable cluster from day one, even running as a single node. I didn't want to tear down and rebuild the moment box two shows up.
2. Services on the cluster need to be reachable from the LAN, not just from inside the cluster. A Grafana pod that only answers on a ClusterIP is useless to every other device in the house.
3. Those services need locally resolvable DNS names. curl 10.9.7.202 is not an acceptable long-term way to reach anything. curl grafana.k8s.myhomenetwork.local is.

Those three requirements shaped every decision that followed, including two I had to walk back mid-build.

Tearing down what was there

Before any of this, the box had a live docker-compose stack running: the observability pipeline from Part 4, two months of uptime, ingesting OPNsense's syslog and NetFlow. First step was backing up the configs (compose file, Vector/Loki/Prometheus configs, Grafana provisioning) into the infra repo for reference. Then docker compose down -v and a full image prune, which reclaimed about 8GB. The configs are sitting in the repo now, a reference for rebuilding the same pipeline as Kubernetes manifests later. That's a separate post.

While I was in there, I also found the box had a second network path that had never been wired up: two onboard 2.5GbE ports, both unused, running on WiFi only. Fixed that too. Plugged in the cable, added a static DHCP reservation in OPNsense, and pointed everything at the wired IP. Not strictly a Kubernetes problem, but a control-plane API server has no business living on WiFi when there's a wire six feet away.

kubeadm, not k3s

k3s is a single binary, bundles containerd, ships with Traefik and a basic LoadBalancer implementation out of the box, and it's fully CNCF-conformant Kubernetes underneath, not a stripped-down toy version. I went with kubeadm anyway. I was about to hand-pick every piece of the networking stack myself (CNI, LoadBalancer, DNS), so I wanted the standard bootstrapping path with nothing swapped out from under me by default. Vanilla kubeadm init it was.

Node prep before kubeadm init was the usual list: swap off, overlay and br_netfilter kernel modules loaded, the bridge-netfilter and ip-forward sysctls set. The one wrinkle was that Docker was already installed on the box, so containerd was already running, just with its CRI plugin disabled, since Docker doesn't need it. Regenerating containerd's config with the CRI plugin re-enabled and SystemdCgroup = true fixed that. Docker and the Kubernetes CRI workloads now share the same containerd binary in separate namespaces (moby and k8s.io) without stepping on each other.

CNI was Cilium, with kube-proxy replacement turned on. The eBPF dataplane, and Hubble's actual visibility into what's crossing the wire, felt like the right tradeoff for a cluster I plan to watch, not just run.

Node came up, control-plane taint removed so the single node can also schedule pods (server and worker, for now), and that's the easy 80%.

LoadBalancer IPs: where MetalLB quietly failed

The second requirement, LAN-reachable services, meant something needed to hand out real LAN IPs to type: LoadBalancer Services. MetalLB is the default answer here, so I installed it, configured an IPAddressPool and L2Advertisement over the free tail of the LAN subnet, and deployed a test nginx Service.

MetalLB assigned an IP. Cilium's own service table knew about it too: cilium-dbg service list showed the LoadBalancer entry pointing at the right pod. And it was completely unreachable. arp -a on my laptop just showed (incomplete). No ARP reply, ever.

The actual failure mode is subtle. MetalLB's L2 mode answers ARP requests for the VIP using a raw AF_PACKET socket on the physical interface. Cilium's eBPF programs, attached to that same physical interface for kube-proxy replacement, sit in the packet path before that raw socket gets a chance to respond. The ARP request never surfaces to MetalLB's speaker. Nothing errors. Nothing logs a conflict. The IP is allocated, the service is healthy, and it's a black hole from the LAN's perspective.

The fix was dropping MetalLB entirely and using Cilium's own L2 Announcements plus LB-IPAM, which exists specifically for this scenario: one eBPF program handles both the service NAT and the ARP reply, so there's no race between two systems fighting over the same NIC. CiliumLoadBalancerIPPool for the address range, CiliumL2AnnouncementPolicy for which interface announces. Reinstalled the test service, and this time arp -a resolved and curl returned 200.

Lesson: running Cilium with kube-proxy replacement and MetalLB side by side is a known-bad combination, but nothing in either project's install output tells you that. You find out from a laptop that can't ARP.

DNS: the part that took two abandoned open-source projects to get right

The third requirement was LAN name resolution. The obvious tool for "give every Ingress/Service in the cluster a DNS name automatically" is ori-edge/k8s_gateway, a single Helm chart, in-cluster CoreDNS variant that watches Kubernetes objects and serves DNS for them directly, exposed via a LoadBalancer IP. Exactly the shape I wanted.

Its container image doesn't exist anymore. The quay.io repository returns an empty tag list and a 401 on pull. Last GitHub release was November 2023. Dead project, and not obviously so until you actually try to pull the image.

Rather than depend on something abandoned for core infrastructure, I rebuilt the same capability from three actively-maintained pieces: a small etcd instance as a record store, CoreDNS configured with the etcd plugin to serve k8s.myhomenetwork.local from it, and external-dns watching Services and Ingresses to keep etcd in sync. More YAML than one Helm chart, but every piece of it ships releases.

Getting external-dns talking to etcd took two wrong turns. First, the CLI flag I expected, --coredns-etcd-endpoints, doesn't exist. That provider takes its etcd connection from an ETCD_URLS environment variable instead, a holdover from the old skydns-era client library it wraps. Second, the pod crash-looped with failed to sync *v1.EndpointSlice after 1m0s: context deadline exceeded, which turned out to be RBAC. I'd granted list/watch on the legacy endpoints resource but not discovery.k8s.io/endpointslices, which newer external-dns versions watch instead. Neither failure mode points at its actual cause in the error text; both took reading source and logs closely to place.

With both fixed, the chain worked: annotate a Service with external-dns.alpha.kubernetes.io/hostname, external-dns writes an A record into etcd within about a minute, CoreDNS serves it. I pinned the CoreDNS LoadBalancer IP with Cilium's io.cilium/lb-ipam-ips annotation so it can't drift out from under whatever's forwarding to it.

The router side wasn't obvious either

Getting queries from an actual LAN device to that in-cluster CoreDNS meant configuring OPNsense to forward the k8s.myhomenetwork.local zone. I expected "Domain Overrides" under Unbound. That's the feature name in every OPNsense guide and screenshot from the last several years, and it doesn't exist in the current UI. What I found first, under Unbound → Overrides, was Host Overrides, which only maps one exact name to an IP, not a subdomain, so it silently didn't help. The actual feature had moved: Services → Unbound DNS → Query Forwarding → Custom forwarding, where you can forward an entire domain, not just one host, to a specific server and port.

Once that was in place, I tested from my laptop against OPNsense itself, not against the CoreDNS IP directly. dig against 10.9.7.1 (the router) resolved a test hostname correctly, and curl against the name returned the actual page. That's the test that matters: proving the path a normal device on the network actually takes, not just that the cluster-side pieces work in isolation.

What's running now

A podinfo deployment, reachable at podinfo.k8s.myhomenetwork.local from any device on the LAN, resolved through OPNsense with no manual host-file entries anywhere. First choice was kuard, the classic Kubernetes demo app, but its gcr.io image is also gone, for the same reason as k8s_gateway: an old public registry that quietly stopped serving anonymous pulls. Two dead upstream images in one afternoon was enough of a pattern to write down. Check that a demo image's registry is actually alive before building a deployment around it.

Everything (manifests, access notes, the reasoning behind each swapped-out component) is checked into a private infra repo, one directory per physical machine. mf1 is machine one. The MetalLB and k8s_gateway detours are documented in place, not edited out, because the next machine that hits the same wall should find the answer already written down.

What's next

More nodes, whenever the hardware shows up. The whole point of choosing kubeadm over a single-box shortcut was not having to redo this. Istio's deliberately not in yet; there are no real workloads to route between, so no reason to carry the complexity. And the observability stack from Part 4 needs to come back, this time as Kubernetes manifests instead of a docker-compose file, using the configs preserved from the teardown as the starting point.

Next: Part 2: A GPU Node That Comes and Goes covers adding a second node with an actual GPU in it.