The daily report from the monitoring box at my parents' house didn't show up. Neither did the confirmation ping to Healthchecks.io, the dead-man's-switch service that alerts if a scheduled job goes quiet. That absence was the actual alert: no email, then a second system noticing the first one hadn't checked in.

The email that didn't arrive

The cron log had the answer immediately:

send_email failed: [Errno -3] Temporary failure in name resolution
daily_report: email send FAILED, not pinging healthchecks

The script tries to reach its SMTP relay by hostname, fails to resolve it, and correctly refuses to ping the dead-man's switch on a failed send rather than lying about success. Straightforward enough, except resolving a hostname is about as basic as networking gets, and this box's actual internet connection had been fine.

A pattern already seen once this week

A few days earlier, an hourly speed-test cron on the same box had started failing outright, not just returning low numbers. Same error shape: name resolution, not connectivity. Querying the house's own AdGuard instance directly (dig @127.0.0.1 www.speedtest.net) resolved instantly and correctly. Querying through the box's actual configured resolver, 100.100.100.100, returned SERVFAIL.

100.100.100.100 isn't a real DNS server. It's Tailscale's own local DNS proxy, the address every device on a Tailscale network is pointed at so that .ts.net hostnames resolve without a separate lookup service. For any other domain it's supposed to transparently forward to whatever the box would normally use. It had stopped doing that.

Patched around it that time: wrapped the speed-test and email-sending code so they resolve through AdGuard directly instead of trusting the system resolver. Functional, but a bandage on a symptom, not a diagnosis of why Tailscale's own proxy had quietly stopped forwarding anything.

What the boot log actually showed

The house's router, its upstream gateway, and this Pi had all been power-cycled together a few days before, while working through an unrelated issue. That was worth checking against: had this box's tailscaled process been running continuously since then, or had it restarted around the same time the resolver broke?

systemctl status tailscaled gave a clean answer: the process had been running since that exact reboot, uninterrupted, for the whole stretch the resolver had been broken. Whatever went wrong, went wrong once, at boot, and just never recovered.

The boot-time journal made the first part of the failure obvious:

logtail: dial "log.tailscale.com:443" failed: ... network is unreachable
trying bootstrapDNS("derp10.tailscale.com", ...) ... network is unreachable
trying bootstrapDNS("derp1c.tailscale.com", ...) ... network is unreachable
trying bootstrapDNS("derp12c.tailscale.com", ...) ... network is unreachable

Six different fallback servers, all unreachable, all in the first second of the process's life. tailscaled's own systemd unit only declares Wants=network-pre.target, an early synchronization point that fires before any interface actually has an address. It never waits on network-online.target, the one that means an interface has a real IP and a route. On a fast boot with no delay before DHCP completes, that ordering gap costs nothing. On this specific boot, it was enough for tailscaled to start its life believing it had no network at all.

The first fix didn't work

The obvious next step was restarting the service now that the network was, provably, fully up. It came back cleanly, reconnected within seconds, Status: Connected in the process status. The resolver was still returning SERVFAIL for everything outside Tailscale's own domain.

That was the first wrong read: assuming the boot-order race was the whole story, and that giving the process a clean restart with working network already in place would let it self-correct. It didn't, which meant something from that bad boot had gotten written down somewhere, not just held in memory waiting to be retried.

The actual mechanism

The post-restart log carried a more specific line than the first one had:

health(warnable=dns-read-os-config-failed): error: Tailscale failed to
fetch the DNS configuration of your device: exit status 1
dns: resolver: forward: no upstream resolvers set, returning SERVFAIL

Tailscale tries to read the box's own underlying DNS configuration so it knows what to forward non-Tailscale queries to. That read was failing, leaving it with an empty list of upstream servers to forward anything to at all.

/etc/resolv.conf was the actual clue: a plain file, timestamped to the same evening as the reboot, rather than the symlink to /run/systemd/resolve/stub-resolv.conf a normal systemd-resolved setup uses. Without that symlink in place, Tailscale's own detection logic couldn't recognize systemd-resolved as the manager in charge and fell back to guessing a different one: dns: using "openresolv" mode. openresolv is a real, separate DNS-management tool this box doesn't actually have. What it has at /usr/sbin/resolvconf is systemd-resolved's own compatibility shim, built to accept the same command name but not the same flags. Its own --help output says so outright: several flags, -l among them, "will cause the invocation to fail." That's the exact flag Tailscale's openresolv-mode code path uses to read existing configuration. Wrong tool detected, real flag, predictable failure, and an empty resolver list as the permanent result until something restored the symlink.

Needing root I didn't have

Fixing either the missing symlink or the systemd unit's dependency ordering needed root. This particular box didn't have passwordless sudo configured for the account being used remotely, unlike the other machine in this homelab. Every prior fix this session had gone through container-based workarounds instead: bind-mounting a directory into a throwaway Docker container running as root, since the account in question was at least in the docker group.

That workaround has a real limit. It gets you file writes and chowns, because the container's root can touch anything reachable through the bind mount. It does not get you the ability to send a signal to a process it doesn't own, even with the host's process namespace shared in: an attempt to restart an unrelated stuck service that way earlier in the week failed with a flat Permission denied, container root or not. Editing a systemd unit and reloading a daemon needed the real thing. Passwordless sudo went into /etc/sudoers.d/, validated with visudo -cf before trusting it, confirmed working with a plain sudo -n whoami.

Scheduling my own rescue

The account now had root, on the exact box whose only working access path ran through the exact DNS proxy that was about to get touched. Every command from here on was a command run over Tailscale, to fix Tailscale, on a machine with nobody standing in front of it.

Before changing anything: sudo shutdown -r +8 "safety-net: auto-revert if DNS experiment breaks access". Then a cancel, to confirm the cancel command itself worked before it might be needed for real. Then the schedule re-armed. Eight minutes, after which the box would reboot on its own regardless of what state the DNS fix left it in, restoring whatever configuration was actually on disk without anyone needing to find the parents' house router and pull a cable.

The actual fix was small: replace the stale /etc/resolv.conf with a proper symlink to the systemd-resolved stub file, restart tailscaled so its detection logic would run again with the symlink in place this time.

The safety net firing for real

The restart command returned, the service reported active. The next command, checking whether resolution actually worked now, timed out. SSH to the box's Tailscale address gave Operation timed out.

That was the exact failure this whole exercise had been built to survive. No further commands, no attempts to route around it some other way. Just waiting, then retrying the same SSH connection with a longer timeout.

It connected. date on the far end read several minutes past the scheduled reboot time. uptime -s confirmed it: the box had rebooted on its own, right around when it was told to, and come back up with working Tailscale connectivity, exactly as the safety net was built to guarantee regardless of whatever the DNS change had actually done.

What came back

DNS worked, checked two ways: nslookup mail.msgdat.in through the box's default resolver returned a real address, and querying 100.100.100.100 directly, Tailscale's proxy specifically, returned the same answer instead of SERVFAIL.

The stronger proof wasn't that it worked. It was that it worked after a full reboot in between, not just inside the live session where the fix had been applied. A fix that only holds until the next power cycle isn't a fix, on a box that gets power-cycled by people who aren't going to SSH in and check.

The containers that were supposed to stay stopped had stayed stopped. The cron schedule was intact. What still isn't answered: why that particular Friday reboot left /etc/resolv.conf as a stale plain file in the first place, rather than the symlink every boot before it had apparently produced correctly. One clean reproduction of a boot-order bug is a fix. It isn't yet an explanation for why this one boot, out of however many before it, went sideways.