This is part 6 of a series on building a homelab Kubernetes cluster. Part 5: Locked Down, and Wired to Tell Me covers the webhook receiver this post's automation ends up deploying.

It started as a small question: why is there an open pull request on a package I maintain. Dependabot had opened it, a major version bump of Express, and it was just sitting there. The answer turned out to have almost nothing to do with Express, and by the end I'd touched a real concurrency bug in a dependency, a broken release pipeline, and the fact that a webhook receiver already running in the cluster hadn't actually been receiving any of the fixes I thought it had.

A six-hour hang that wasn't the dependency's fault

The PR's check had failed after running for six hours, the default job cap on GitHub Actions, cancelled rather than failed outright. That's not what a real test failure looks like. A real failure is fast. Six hours means something hung and nothing was watching for it.

The fix for that part was mechanical: a per-test timeout so a stuck test fails in seconds with a clear error instead of silently eating the job's entire budget, plus a job-level cap as a backstop. Small change, and it immediately paid for itself, because the very next run caught something real. A websocket test kept timing out waiting for an event that never arrived, reproducing reliably in CI and never once locally. That mismatch is the tell that it's environment-specific, not flaky in the usual sense.

The actual bug lived in a small file-watching dependency the package used to tail a log file. Its change handler stored the in-progress read's file descriptor on a shared field, and two file-change events landing close together, routine under Linux's inotify but rarer under macOS's file events, let the second event overwrite that field before the first read's callback finished. The first callback then closed the wrong file descriptor, that read failed, and the error handler swallowed it silently, dropping a line that was supposed to stream to a connected client. Small package, real bug, and it had nothing to do with the Express bump that started this. It just happened to be the PR unlucky enough to hit it first.

Replacing that dependency with a small in-house version fixed it outright, verified over repeated runs both locally and in CI where it had failed almost every time before.

Fixing a bug well enough to trust the next one

With that fixed, there was still a policy question sitting underneath it: major version bumps were deliberately excluded from auto-merge, on the reasoning that a breaking change deserved a human look. Worth asking directly what a human would actually catch there that a good enough test couldn't.

The honest answer, for this package specifically, was one real gap: every existing test built both the web server and the router being tested from the same installed version of Express, because there was only ever one installed. The package's actual job is being mounted into someone else's Express app, possibly running a different major version than the one bundled inside it, and nothing tested that mismatch at all. An npm alias made it possible to install a second major version of Express purely for testing, build a host app on it, and mount the package's router into that mismatched host. It passed. With that specific gap closed, the case for holding every major bump for a human evaporated, and the policy changed to match: any dependency bump merges once its tests pass, full stop.

Merged, and nothing happened

Express 5 landed on main. Nothing published. This is the part that actually should have been obvious sooner: the release pipeline only ran when the package's version field itself changed by hand, and nobody had touched it. Every fix from the last several days had merged and gone precisely nowhere.

The fix was switching to an automated release tool that reads commit types (a fix versus a feature versus a breaking change) and handles the version bump, the publish, and the changelog itself, no human editing a number ever again. Getting there took two more real bugs, both from tooling rather than anything in the package: the release tool's current version needed a newer Node than the job was running, and separately, its own built-in npm-publishing step failed its authentication check against a publishing method that the plain npm publish command, run directly, already handled correctly in the exact same job. Routing the actual publish through the command that already worked, instead of the tool's own broken path to the same place, closed it. One manual version bump shipped everything that had piled up, and every release since has been automatic.

The fix that still wasn't running anywhere

All of that fixed how the package gets published. It said nothing about the app already running in the cluster that depends on it, a small webhook receiver and live log viewer, the same one Part 5 covers moving behind a Cloudflare Tunnel. Checking turned up two separate gaps. Its lockfile was still pinned to an older version, because the automated dependency bump for it hadn't run yet on its own weekly schedule. And even once that bump landed, nothing would rebuild or redeploy anything, because no such pipeline existed yet. The pod had been running whatever image was built by hand, once, and would keep running it indefinitely.

Building that pipeline meant deciding where it could actually run from. The private registry this cluster uses lives on LAN-only DNS, and the cluster's own API server isn't reachable from outside it either, so a GitHub-hosted runner was never going to reach either one. The fix was a small self-hosted runner living as a pod inside the cluster itself, the only thing with a legitimate path to both, with just enough permission granted to it to restart one specific deployment and nothing else. Wired to trigger once the app's own tests pass on its main branch, it builds the image, pushes it, and rolls out the new pod, the same three steps a person would otherwise do by hand every time.

Proof, not assumption

Bumping the dependency by hand rather than waiting on next week's schedule, the whole chain ran once, live: a commit landed, tests passed, the image built and pushed, and a new pod came up with a different image digest than the one before it. A curl against the public endpoint, the same one anyone on the internet can reach, landed in the running app's log file and streamed live to a browser watching the viewer, a small proof that the entire path, from a dependency fix to a pod serving real traffic, actually still works end to end.

What this adds up to

None of this started as a project. It started as one open pull request that shouldn't have still been open, and pulling on it kept surfacing the next thing that was quietly not working: a test suite that couldn't fail fast, a real bug hiding behind a policy that existed to guard against a different, less likely one, a release pipeline that had silently stopped doing its one job, and a running application that had no way to ever find out a fix existed at all. Each of those was invisible until something forced a look. What's left is a chain with no missing link left in it: a fix merges, gets tested, gets published, and runs in front of real traffic, and the only thing a person has to do anywhere in that path now is write the fix itself.