This is part 4 of a five-part series. Part 1: The Rabbit Hole covers the investigation, Part 2: Building an IT Department covers what I built because of it, Part 3: The Bill for Letting AI Watch the House covers what running it for real actually cost, Part 5: What a Router's Own Name Gave Away covers a two-city detour hiding in a hostname.

This one starts with a fix that worked exactly as intended and quietly broke something else on the way in. Nobody noticed for a full day, because the thing it broke, a DNS fallback, only matters during an outage.

The fix that started it

The house's DNS resolver had been averaging around 400 milliseconds to answer a query, which is slow enough to be its own small mystery. The cause turned out to be almost funny in retrospect: it was configured with two upstream resolvers, load-balanced, splitting every query between them. Splitting traffic between two persistent encrypted connections meant neither connection got used often enough to actually stay warm, so a good fraction of queries were paying the cost of a fresh handshake, every time. The fix was obvious once found: one upstream, kept warm by constant use. Average response time dropped by roughly an order of magnitude.

What that fix also did, quietly, was remove the fallback. One upstream configured means one upstream, full stop. If it goes down, there was nothing else to try.

Diagnosing the failed resolution

The next day, a routine check on something unrelated (a personal dynamic-DNS address, the kind of thing you point at your own home network to reach it from outside) failed to resolve. Tried again a few minutes later. Worked fine. The instinctive read is "internet's just flaky sometimes," and most people would leave it there. That's how this kind of bug survives: nobody looks closer because nothing forces them to.

The resolver's own log had a matching entry at the exact timestamp of the failure: an attempt to reach its one configured upstream, failed, with a network-level error that reads like a dropped connection rather than a clean rejection. Not an isolated failure either: the log had hundreds of the same error, clustered in bursts, spread across the better part of a day. The upstream provider itself was having a rough patch, and the resolver had nothing else to fall back on.

Why nobody downstream noticed sooner comes down to one specific, unintuitive mechanism. A phone on this network gets handed two DNS servers by the router: a primary, and a secondary meant as exactly this kind of safety net. The natural assumption is that if the primary goes down, the phone quietly switches to the secondary and nothing ever visibly breaks. That assumption is wrong: the resolver never actually went down. It stayed up, stayed reachable, and kept answering every query, it just started answering with "I tried, and failed" instead of a real address. A prompt error response and a genuine timeout look completely different to the resolver returning it, but they're indistinguishable to a router deciding whether to try the backup: both a real answer and a valid-but-useless one are, from that router's point of view, "the primary DNS server responded." Nothing about that ever triggers the router's own fallback logic, because that logic is built to detect an unreachable server, not a server that's alive and simply passing along someone else's bad news.

Adding a tested fallback

The direct fix, a proper fallback list on the resolver itself instead of relying on the router's blunter logic, was straightforward to add. What wasn't straightforward, and what actually mattered, was the decision not to just add it and move on. A fallback that's never been tested against a real failure is a fallback you're hoping works, not one you know works. That gap, hoping vs. knowing, is what let this bug hide for a full day in the first place.

So: a deliberate test. A temporary firewall rule on the house's own monitoring box, dropping every packet headed to the DNS provider that had been failing, simulating exactly the kind of outage that had actually happened, on demand, with full control over exactly when it started and stopped. The rule itself is one line. The part worth explaining is the line right next to it, which exists specifically because deliberately breaking a real household's internet, even briefly, deserves care: a shell trap, registered before the block goes in, that guarantees the rule gets removed the instant the script exits, for any reason, success, failure, an interrupted command, anything. It's the same idea as a finally block in most programming languages: whatever else happens, this cleanup runs. The very first test attempt actually did fail partway through, and the block came down automatically anyway, before there was ever a moment to notice or intervene. That's what made it reasonable to run this test on a real network at all: the failure mode of "something goes wrong mid-test" was already handled, not something being trusted to human attentiveness at four in the morning.

What the test actually found

The fallback worked: with the primary blocked, queries came back with answers from the backup resolvers, confirmed directly in the logs, not inferred.

The part that wasn't as clean: it took over thirty seconds to get that answer. Three separate test lookups got no response at all within twenty seconds and simply gave up. A fallback that takes half a minute to kick in isn't really a safety net for anything interactive. A phone, a browser, any app loading something during that window would just look broken: functionally no different from having no fallback at all.

The instinct was to reorder the fallback list, on the theory that the first backup being tried was somehow slower to fail over to than it should be. That instinct was wrong, or at least incomplete: reordering it changed nothing, worst case stayed at roughly the same thirty seconds. The resolver's own logs during the test told a more useful story than the theory did: two different backup providers had both successfully answered different queries within the same second, which reads much more like the resolver racing several attempts at once than working through a strict priority list. Rather than keep guessing at an internal retry mechanism from the outside, the more reliable fix was to control the one thing that mattered regardless of the exact mechanism: how long any single attempt is allowed to hang before being abandoned. Turning that down directly bounds the worst case, no matter how the resolver internally decides what to try and in what order.

Tightening that timeout and re-running the exact same test, repeatedly, across a handful of different domains each time, produced a clean, consistent result: worst case dropped from thirty-plus seconds to right around four. Verified the same way every other number in this post got verified, not read off a settings page and trusted: broken deliberately, measured, fixed, broken again to confirm the fix actually held.

Automating detection

The whole reason this took a day to surface is that nothing was watching for it. That felt like the actual bug, more than any specific configuration mistake: a failure mode that only gets caught if a human happens to hit it personally and happens to go looking. So the last piece wasn't a network fix at all, it was making sure the daily report reads the resolver's own error log going forward, counts failures, and flags a pattern automatically. Since an outage at one provider tends to show up two ways at once, both as resolver errors and as a jump in plain network latency to that same address, the report now checks both together and says which kind of problem it's looking at: a path-level outage reaching that specific provider, or something narrower and more local. That's the difference between "something upstream is having a bad day" and "something here is misconfigured;" working it out by hand took an evening this time. Next time, it should just be in the email.

Summary

The gap wasn't really about DNS. It was between a fix that looks complete and a fix that's been shown to survive the failure it's supposedly guarding against. The fallback existed the whole time this was "fixed;" it just hadn't been tested, which for a system nobody's watching around the clock is functionally the same as it not existing.