This is part 3 of a series on building a homelab Kubernetes cluster. Part 1: One Box, Real DNS covers the first node. Part 2: A GPU Node That Comes and Goes covers the second.

Every internal service in this cluster lived under a made-up domain, something that looked exactly like the kind of placeholder you'd invent for a homelab and never think about again. Then Chrome flagged it as a dangerous site. It turned out to be a real, previously-registered domain with an actual bad reputation attached, not something I'd invented at all. This is the writeup of fixing that properly: moving everything to a subdomain I actually control, and getting real, browser-trusted certificates for hostnames that are never reachable from the public internet.

The problem, once it was visible

A domain I didn't own, with a history I knew nothing about, was the base for every internal hostname in the cluster. That's a bad foundation for two independent reasons. First, the obvious one: it wasn't mine, and its reputation wasn't under my control. Second, less obvious until it's spelled out: even if it had been a clean, unused domain, everything was still running on plain HTTP internally, since there was no realistic path to TLS for hostnames nobody outside the LAN can reach.

The fix for both turned out to be the same move.

Reusing a domain I already own

This blog runs on abhijeetapsunde.com, hosted on Cloudflare. Reusing a subdomain of it, k8s.abhijeetapsunde.com, for everything internal solves the ownership problem outright. It also unlocks the certificate problem, because Cloudflare's DNS API makes something possible that a domain I didn't control never could have: proving ownership without ever exposing anything publicly.

Let's Encrypt's usual challenge, HTTP-01, needs a URL the certificate authority can actually reach. None of these hostnames are reachable from outside the LAN, by design, so that path is closed. DNS-01 doesn't have that requirement. It only asks for the ability to create a specific TXT record via the domain's DNS API, entirely independent of whether anything is publicly resolvable. cert-manager, running in the cluster, can request a real wildcard certificate for *.k8s.abhijeetapsunde.com, prove it via Cloudflare's API, and get back something every browser trusts by default, no internal CA, no manual trust-store installs on every device in the house.

Setting it up

cert-manager installed cleanly from its official manifest. The interesting part was the ClusterIssuer configuration: a Cloudflare API token scoped to Zone.DNS: Edit on just that one zone, stored as a Kubernetes Secret (never written to a file, never committed anywhere), referenced by two issuers, one pointed at Let's Encrypt's staging environment, one at production.

Staging first, deliberately. Production Let's Encrypt has real rate limits, five duplicate certificates per exact hostname set per week, and a first attempt at any new ACME setup is exactly the kind of thing that goes wrong in a small, fixable way. Staging doesn't count against that limit. A test wildcard certificate against staging validated the whole DNS-01 flow end to end in under a minute: token permissions, Cloudflare API access, the TXT record round-trip, all confirmed working before touching production at all.

Two SANs, one TXT name, a genuine collision

The production wildcard request stalled for over twenty minutes, long enough that it was worth digging into why rather than waiting it out. Querying Cloudflare's own authoritative nameservers directly for the challenge TXT record kept returning a clean, unambiguous NOERROR with zero answers, a real negative response, not a caching artifact. Cloudflare's own API said something different: the record existed, correctly named, correct content, created minutes earlier. Token permissions checked out too. A record genuinely present in Cloudflare's control plane and genuinely absent from its own authoritative answers is not a normal state, and it was worth finding the actual reason instead of assuming propagation lag and waiting longer.

The Certificate had asked for two SANs: the wildcard, *.k8s.abhijeetapsunde.com, and the bare apex, k8s.abhijeetapsunde.com, mostly out of habit, since nothing is actually served at the apex. Both need a DNS-01 challenge, and both challenges need a TXT record at the exact same name, _acme-challenge.k8s.abhijeetapsunde.com, each with a different required value. cert-manager created one of them. The other challenge sat there the entire time with zero events logged against it, never even attempted, silently blocking the whole certificate from issuing while the visible symptom, the wildcard challenge's propagation check, looked like ordinary DNS lag.

Dropping the apex from dnsNames and requesting only the wildcard removed the collision outright. The certificate issued within a minute of that change.

Worth remembering for next time: when a Certificate requests both a wildcard and its own bare apex, check both challenges individually before assuming a slow one is just slow. A stuck DNS-01 challenge for a name that collides with another active challenge looks identical to propagation lag until you check the other challenge's own event log and find nothing there at all.

Bringing in Istio

The wildcard certificate is only useful if something actually terminates TLS with it, and that's an Ingress controller's job. Istio had been deliberately deferred until there were real workloads to route between, on the theory that a service mesh's complexity shouldn't be paid for before it's earned. There are real workloads now: n8n, Ollama, a demo app. This was the trigger to bring it in for real rather than reaching for something lighter as a stopgap and replacing it later.

One thing worth knowing before installing Istio next to Cilium, because it's a documented interaction and not a mystery worth rediscovering the hard way: Cilium's eBPF socket-level load balancing, on by default with kube-proxy replacement, can bypass the iptables rules that sidecar-based service meshes rely on to intercept pod traffic. Cilium's own documentation recommends socketLB.hostNamespaceOnly=true specifically for this scenario. Applying that before installing Istio, rather than after debugging a silent traffic problem, was the one preventive step that actually paid off this time instead of becoming its own investigation.

Istio's default profile installed cleanly after that: istiod, an ingress gateway, one shared LoadBalancer IP for everything HTTP(S), pinned the same way the DNS server's IP was pinned in Part 1, so it can't drift under whatever forwards to it.

Rewiring everything that already existed

Before Istio, n8n, Ollama, and the podinfo demo each had their own dedicated LoadBalancer IP. That's the wrong shape once a shared ingress gateway exists: the normal pattern is one gateway IP, with Gateway and VirtualService resources routing by hostname to the right backend internally. All three moved from LoadBalancer Services to ClusterIP, each gained a VirtualService referencing the shared Gateway, and the Gateway itself holds the wildcard certificate for TLS termination.

external-dns needed one change to keep up: adding istio-virtualservice as a source alongside the existing service and ingress sources. It resolves each VirtualService's hostname to the ingress gateway's Service automatically, so DNS for every app updated correctly without hand-editing a single record. The one thing that stayed exactly as it was: the DNS server itself keeps its own dedicated LoadBalancer IP, since it's a UDP service, not HTTP, and has no business going through an HTTP-terminating gateway.

Last piece: OPNsense's Query Forwarding entry moved from the old domain to k8s.abhijeetapsunde.com, same mechanism as before, new zone name.

Two smaller gotchas from the same afternoon

Neither is dramatic on its own, both are worth writing down because each one looked like something else at first.

CoreDNS doesn't hot-reload its Corefile from a ConfigMap change. Updating the zone name from the old domain to the new one and applying it left the running pod serving the old zone entirely, REFUSED on every query for the new domain, until the Deployment was explicitly restarted to actually pick up the new ConfigMap. Obvious in hindsight, a ConfigMap update alone doesn't restart anything that mounted it.

And a purely self-inflicted one: copying two files that happened to share the same name, virtualservice.yaml, from two different app directories to one shared remote /tmp/ in a single batch. The second copy silently overwrote the first before either got applied, so one VirtualService quietly never existed at all. Batch file transfers to a shared destination need unique names, not just unique source paths.

Confirmed working, the whole chain

Once OPNsense's forwarding entry pointed at the new domain, the full path checked out end to end: curl https://n8n.k8s.abhijeetapsunde.com resolved through OPNsense, hit CoreDNS, landed on the Istio gateway, and came back with a certificate a browser accepts without complaint, issuer: Let's Encrypt, verified, no -k flag required, HTTP/2 200. Every hostname under *.k8s.abhijeetapsunde.com gets that automatically now, routed through one shared gateway instead of a pile of individual LoadBalancer IPs. Whatever gets deployed next just needs a VirtualService, no new DNS work, no new certificate request, the wildcard already covers it.

Next: Part 4: Self-Sufficiency, Not Self-Signed takes that same wildcard cert somewhere the cluster doesn't reach at all.