The complaint was vague in the way home network complaints usually are. A phone and a tablet lost connectivity for a moment. Things came back. Nothing was obviously broken and every dashboard was green.

In most houses that is the end of it. The wifi stopped, the wifi came back, nobody files a ticket against their own living room.

That is not how this one runs. If something broke, I want to know what broke, and "it fixed itself" is not an answer, it is the absence of one. Getting an answer means having the data to produce one before the next time, which in practice means more probes, more logs, more alerts, and something sitting on top of all of it that can read a few hundred thousand log lines and tell me what changed. Most of what follows is that apparatus being built one gap at a time, because each thing I added revealed the next thing I could not see.

Why nothing showed up

DNS here runs on an OPNsense box: Unbound as the resolver, forwarding over DNS-over-TLS to Quad9, with a firewall rule that redirects all LAN port 53 traffic to it so no device can use an outside resolver by accident or design.

The health probe for it ran every 60 seconds and asked one question: does a name resolve right now. The outages, once I could finally measure them, lasted about 15 seconds.

A once-a-minute sampler catches a 15 second event roughly a quarter of the time. The probe was not broken, it was the wrong shape. A liveness probe samples. What I needed was something that integrates, so an event gets recorded whether or not anyone was looking when it happened.

The replacement is a counter built on logs rather than on a live check. OPNsense syslog already ships to Loki, so a script queries it every minute and writes the result out for Prometheus:

sum(count_over_time({job="opnsense", appname="unbound", severity="3"}
  |= "SERVFAIL" != "arpa" [5m]))

The != "arpa" matters more than it looks. Apple devices generate thousands of reverse lookup and mDNS failures a day against private address space. Left in, they bury everything else and make the whole severity channel useless.

The fix that hid the problem

Both DoT forwarders were Quad9. One operator, two addresses, no independent fallback. When Quad9 was unreachable both were unreachable, and forward-addr in Unbound has no concept of ordering: it selects by round trip time, so a second entry is not a backup, it is a coin flip.

What Unbound does have is forward-first. Turn it on and a failed forwarder lookup falls back to recursing from the root servers instead of returning SERVFAIL. Enabling it fixed the user visible symptom immediately.

It also made the SERVFAIL counter I had just built go quiet, because queries now succeeded. The signal disappeared the moment the fault stopped causing visible damage, which is a bad property for a signal to have.

So the counter moved again, onto the thing that only happens during fallback. Normally every DNS query leaves this firewall as DoT on port 853 and plain outbound port 53 is close to never used. Over one six hour window: 10,102 packets to 853, zero to 53. A burst of outbound port 53 therefore means Unbound gave up on the forwarders and went to the roots.

That counter has to be scoped to the WAN interfaces. My first version matched any outbound port 53 and immediately picked up the firewall querying in-cluster CoreDNS for a forward zone, which is internal traffic with nothing to do with recursion.

Two events, two shapes

The alert fired for the first time on a Monday morning, and the event was not the one it was built for.

A client asked for an HTTPS record on a vendor's SSO login host, a long CNAME chain ending in a DNSSEC-signed Okta zone. Quad9 returned nothing usable, forward-first did its per-query job, and Unbound walked the delegation itself: root servers, then gTLD servers, then the CDN, then the zone's authoritative nameservers, which it retried four or five times each with widening backoff before giving up 81 seconds later with:

SERVFAIL <login.example-vendor.com. HTTPS IN>: exceeded the maximum
nameserver nxdomains

That error comes from the iterator walking a delegation, not from the forwarding path, so it is itself evidence that recursion was running.

Total cost: 33 packets, 11 distinct destinations, one failed query. The resolver health metric never moved. One broken domain somewhere on the internet, which happens most days.

The second event, later the same day, looked different in a way that mattered:

169 packets, 100 distinct destinations, 15.2 seconds
198.41.0.4     a.root-servers.net
170.247.170.2  b.root-servers.net
192.203.230.10 e.root-servers.net
199.7.83.42    l.root-servers.net
202.12.27.33   m.root-servers.net
... plus most of the gTLD servers and several large authoritative operators

Nearly the entire root server set contacted at once, with 57 packets in a single second. That is Unbound rebuilding its delegation chain from cold for many unrelated names simultaneously, which is what losing the forwarders looks like.

The two shapes separate cleanly, and not on volume:

	packets	destinations	per destination
One bad domain	33	11	3.0, narrow and deep
Forwarder failure	169	100	1.7, wide and shallow

Deep means the same few nameservers retried. Shallow means one or two queries each to many unrelated servers, which only happens when the forwarder is gone. Destination spread is the discriminator. Packet count is not: the serious event was only five times the size of the trivial one.

An alert that could not fire

The second event did not page me, and that was my bug.

The metric is a five minute rolling window. An instantaneous burst keeps it elevated for at most five minutes and then rolls out. I had put for: 5m on the alert rule on top of that.

20:19:22-20:19:37   the actual 15 second burst
20:20  spread=100   condition true
20:21  spread=100   condition true
20:22  spread=100   condition true
20:23  spread=100   condition true
20:24  spread=100   condition true
20:25  spread=0     rolled out, duration never satisfied

The rule raced its own window and lost. Any for: at or near the length of the metric's own window is unsatisfiable for exactly the short sharp events that matter most. Both rules are now for: 0m, because the window already is the smoothing.

Probing by layer

Knowing the forwarder path failed is not the same as knowing why. TLS handshake failure, reachability loss, and the resolver itself misbehaving are three different faults with three different owners, and nothing I had could tell them apart. Unbound at verbosity 1 logs no TLS diagnostics at all. A Loki search for tls|ssl|handshake|certificate returned 14,774 hits, every one of them a domain name like ssl.gstatic.com.

The answer was a probe that tests each layer separately: ICMP, then TCP on 853, then a TLS handshake with certificate verification, then a real DoT query. It did not work on the first attempt, for a reason worth repeating. The firewall blocks outbound 853 from the LAN, so nc -z 9.9.9.9 853 from the probe host failed while ICMP succeeded. Checking that before writing the probe is what stopped me shipping something aimed at a port it could never reach.

With a pass rule added for the probe host specifically, all four layers became measurable. The rule had to go on the right interface, and my first attempt did not. The UI happily showed my pass rule sitting above the block rule while the port stayed shut, because the interface I picked was not the one the probe host sits on. The firewall log said so directly:

rule=129 block 198.51.100.10:55698 -> 9.9.9.9:853 flags=S

Asking the firewall which rule dropped the packet beat reasoning about rule ordering in the UI.

Sampling, again

The 60 second probe had the same flaw as the original health check: the events are 15 seconds long. So the probe became a service that checks TCP 853 every 10 seconds and accumulates failures into counters. A two second outage at 04:00 is still visible at 05:00 regardless of when Prometheus scraped. On failure it immediately runs the full TLS diagnostic and writes a timestamped record, which captures the detail while the fault is still happening instead of afterwards.

That caught the next one in the act:

14:54:04  forwarder A  tcp=FAIL
14:54:19  forwarder B  tcp=FAIL
14:54:44  forwarder A  tcp=FAIL

With the layers separated the attribution was unambiguous. WAN gateways: 0% loss, round trip time flat at 2.7ms. ICMP to both Quad9 addresses: down. TCP 853 to both: failed. Client facing DNS: never dropped, because forward-first absorbed it again, 819 packets across 206 destinations and exactly one public SERVFAIL.

Both anycast addresses unreachable at once while our own gateways were clean. Not the cable, not the firewall, not TLS.

The upstream problem

Underneath all of this sits a limitation in Unbound itself. forward-addr takes a list of forwarders and picks between them by round trip time. There is no way to say "use this one, and only use that one when the first is gone". For a setup that picks a resolver deliberately, for its filtering or its privacy policy, RTT selection quietly overrides the choice.

forward-first is the closest thing available and it falls back to recursion, not to another forwarder. So during an outage the house resolves from the root servers with no filtering at all.

There is an open feature request upstream for exactly this, #1402, asking for sequential forwarder selection. I added a use case to it: a filtering resolver as the primary, with two general purpose resolvers behind it that should only ever be queried when the first is unreachable. Unbound cannot express that today.

So I implemented it. #1537 adds a forward-sequential option to forward-zone: strict ordering, first listed tried first, the next one only when the one above it is unreachable. It is 401 lines across the config parser, the iterator's server selection, the docs, and two testbound replay tests covering the ordered case and the failover case.

It is open, not merged, and the maintainers may well prefer the priority-tier design from a different proposal on the same issue. Either outcome is fine. The configuration layer cannot simulate an ordering the daemon does not have, so this is the place it has to be fixed.

The latency nobody asked about

While checking the path I measured something that had been sitting in plain sight:

8.8.8.8   (Google)      2.4 ms
1.1.1.1   (Cloudflare)  2.8 ms
9.9.9.9   (Quad9)      27.3 ms
149.112.112.112        28.5 ms

Same host, same minute, same WAN. Quad9 is ten times further away than the other two. Traceroute shows the path still at 2.3ms inside the local metro at hop 3, then 29.2ms at hop 6 on the carrier's long haul backbone. Cloudflare exits through a local edge router and completes in 2.1ms.

All three are anycast and Quad9 publishes a presence in this metro. The route was not reaching it.

Unplugging the primary WAN and repeating everything on the cable backup settled it:

	Cloudflare	Google	Quad9
Fiber carrier	2.8 ms	2.4 ms	27.3 ms
Cable carrier	10.5 ms	13.7 ms	11.9 ms

The cable link is slower in general and still reaches Quad9 more than twice as fast, staying inside the local metro the whole way. The penalty is specific to one carrier and one destination.

Peering

I sent the measurements to the fiber carrier's IP admin contact, which is published in ARIN WHOIS. They replied the same day, and the answer was unambiguous.

There is no direct peering with Quad9. The resolvers that answer locally do so because their operators are direct peers or customers of the carrier. Quad9 is not, the carrier does not approach networks of that size about peering, and the thing that would change it is Quad9 buying transit from them. Nothing is misconfigured.

So the traffic goes onto the backbone and meets Quad9's network wherever the two happen to interconnect, which from here is about 27ms away. That is a commercial decision rather than a fault, and no amount of configuration on my side moves it.

It also offers an explanation for the failures and not only the latency. A single out of region interconnect is a single shared dependency, which fits both anycast addresses going unreachable together while Cloudflare, a direct peer, stayed at 0% loss.

Worth saying plainly: the carrier answered a residential question, the same day, with a direct and accurate answer. That is not what I expected when I sent it.

Where it stands

Unresolved: whether the failure rate actually drops on the other carrier. Four hours in, zero failures against a baseline of roughly 0.2 per hour, which sounds convincing and is not. Four hours at that rate would expect about 0.8 failures, so observing zero has close to a 45% chance even if nothing improved. It needs a twelve hour window to match the baseline before it means anything.

Also unresolved: whether to add a second operator as a fallback. The argument for it does not depend on comparing blocklists, which is the comparison I tried to make and got wrong twice.

The first attempt queried two public resolvers from inside the network and got identical answers from both, which I reported as inconclusive. It was not inconclusive, it was invalid: the port 53 redirect means both queries were answered by my own Unbound. Querying a documentation address with nothing behind it answers too, and version.bind from all of them returns unbound 1.26.1. The redirect was doing exactly its job and I had forgotten it was there.

The second mistake was assuming a single domain could say anything about blocklist quality in the first place. It cannot. Coverage, false positive rate and how quickly new threats are added all need a corpus of known-bad and known-good names run against both resolvers at the same moment.

The argument that does hold is narrower. Today, when the primary forwarder is unreachable, forward-first resolves from the root servers with no filtering at all. So during an outage the comparison is not one provider's blocklist against another's, it is any filtering against none.

None of that is finished. What changed is that the next 15 second DNS fault will leave a record of which layer broke, how long it lasted, and which upstream was at fault, instead of a phone that stopped working for a moment and a green dashboard.