This is part 3 of a five-part series. Part 1: The Rabbit Hole covers the investigation, Part 2: Building an IT Department covers what I built because of it, Part 4: Prove the Safety Net Actually Catches You covers a fix that quietly broke something else.

Part two ended with a system deployed and staged, not yet proven. This part covers what happened once it started running for real: the upload cap that started this whole series resolved on its own, for reasons I still don't know, and running the monitoring system against a live house turned out to be mostly about something else: what it actually costs to let an AI agent watch a house every day, and a string of small bugs that only surfaced by actually running the system.

The upload cap resolves on its own

Somewhere around mid-morning, local time, the upload cap that had been sitting at roughly a ninth of download speed since before this series started, the thing that started the whole investigation, just stopped being capped. Upload throughput went from a number that had been essentially flat for days to matching, and at moments exceeding, download. No warning, no visible trigger, no ticket, nothing.

I didn't trust it immediately: two machines involved in checking this were reporting two different local times. So I threw away both machines' clock labels entirely and went back to the one thing that can't lie about time: the raw Unix timestamp Prometheus actually stored the sample under. Converted independently, cross-checked against the dashboard's own rendering once I confirmed which timezone it was actually displaying in, and it held up. The jump was real: it happened within about a five-minute window, and it's been holding steady since.

What didn't change alongside it is the stranger part. The route to a stable anchor address, hashed and checked every fifteen minutes, stayed identical before and after. The upstream identity was unchanged. Nothing in the PPPoE session state suggested a reconnect. Whatever happened, it happened somewhere upstream of everything this system can actually see, which on a franchise-resold connection over a shared fiber tree, per everything in part one, could mean a provisioning change, a shaping profile getting swapped, or anything else happening on hardware I have no visibility into and no name for. Two posts of investigation, and the fix arrived on its own, by a route I still can't identify.

The daily digest agent

Part two ended with a design: a daily Claude-run digest, reading whatever the house's monitoring had collected and writing an actual explanation, not just a metrics dump, judgment over checklist. The idea was that something like the upload cap resolving itself is what a fixed threshold can't narrate. It can tell you a number changed. It can't tell you "this is the fourth day running at this ceiling" or "this looks like the same pattern finally breaking, not a new problem."

Once it was actually running, against a live connection on a real schedule, the first cost showed up somewhere else entirely: judgment isn't free, and "let the AI figure it out every day" is a worse default than it sounds.

What a quiet day cost

The first design had the agent do everything, every single day: read the full history, query every metric fresh, decide what mattered, write the report, decide what model tomorrow's run should use, send the email. All of that, every day, whether or not anything had actually happened.

That turned out to cost real money even on days with nothing to say. Somewhere between a quarter and just over half a dollar per run, for a report whose entire content, on a quiet day, is "nothing changed." Not because the model is expensive to run once. Because a full agent run re-derives the entire world from scratch every time: rereads yesterday's findings so it knows what's already been said, re-queries every metric it might need, reasons about all of it, and writes prose, and every step of that is billed the same whether the answer is "here's a real problem" or "here's nothing."

That's an easy trap to fall into right now, and not just for a house in rural India. Point a capable model at a recurring task, let it run on a schedule, trust it to be reasonable about what's worth flagging: it's the obvious shape for a huge number of "AI agent" products showing up this year, and it has a specific, quiet failure mode. It gets most expensive on the days it has the least to say, because "nothing happened" still costs a full pass of judgment to arrive at.

Splitting fixed checks from AI judgment

The fix wasn't a better prompt. It was admitting that most of what the agent was doing every day didn't need an agent at all.

The redesign splits the work into two tiers. A deterministic script runs every day: it pulls every metric directly, applies a fixed set of threshold and drift rules, exactly the kind of thing a decade of monitoring tooling already knows how to do well, and assembles a plain report. That part costs nothing, because none of it touches a model. Claude only gets invoked at all if one of those fixed rules actually trips, and even then, only with a short, specific prompt: here is exactly what was flagged and why, explain it, not "go figure out what's going on," which is the instruction that made the first design expensive in the first place.

The difference in practice: a day with a real finding worth explaining went from roughly sixty cents to about six, same underlying judgment, now applied to one specific, already-identified fact instead of asked to rediscover the entire state of the house from nothing. AI belongs at the narrow point where something needs explaining, not spread across the whole job by default.

Bugs only a live run surfaced

None of what follows was caught by design, and none of it was caught by testing anything in isolation. All three only showed up because the system was actually running, continuously, against real data, for long enough that something eventually looked wrong to a human paying attention.

A test container flagged critical, every day, for a container that never should have been checked at all. One of the health checks walked every container on the box and flagged anything not currently running. Sensible in principle, except one of the containers on that box was a completely harmless one-shot test image, the kind that prints a message and exits on purpose, left over from an early setup step nearly two years earlier. It had been correctly stopped since the day it ran. The check didn't know the difference between "this thing crashed" and "this thing did exactly what it was built to do," so it quietly flagged the same non-problem as critical every single day until someone actually read a report closely enough to ask why.

Health checks that had been silently wrong since the day they were turned on. Three separate probes meant to confirm DNS was actually working had been reading "failing" on every single sample since the monitoring stack first came up, despite DNS resolving correctly the entire time, verified by hand. The cause had nothing to do with DNS itself: two pieces of the same Docker stack were configured to see the network differently from each other, so a probe checking "can I reach the DNS service on this machine" was, invisibly, checking a completely empty spot instead. Nothing was actually broken. The thing built to prove it was working was checking the wrong address, silently, from day one.

A well-known, off-the-shelf dashboard that rendered completely empty. Not a hand-built panel, an entire pre-made community dashboard covering system health end to end, dropped in because it was already proven and popular. Every single one of its thirty-odd panels showed nothing. Not because any of the underlying data was missing, it was all there and correct, but because every panel on that dashboard was still pointing at an internal reference left over from whichever system it had originally been built and exported on, one that simply didn't exist here. The fix was a single find-and-replace across the whole file.

A fourth, smaller one for anyone who's tried to run anything on a Raspberry Pi: the tool used to measure real throughput needed three different install attempts before one actually worked on this hardware, not because the tool was broken, but because the two normal ways software gets installed on Linux both quietly assumed hardware this box doesn't have.

What the system actually caught

The thing that actually started this series, the upload cap, fixed itself on a schedule I had no control over, for a reason I still don't know. Nothing I built caught it happening or explained why.

What the system actually caught was smaller and less dramatic: a monitoring rule crying wolf about a two-year-old non-problem, health checks that had been quietly lying since the day they were turned on, a dashboard that looked fine in a screenshot and was completely broken, all of it caught while costing, on an average day, close to nothing.