This is part 6 of a series on building a homelab Kubernetes cluster. Part 5: Locked Down, and Wired to Tell Me covers the webhook receiver this post's automation ends up deploying.

It started as a small question: why is there an open pull request on a package I maintain. Dependabot had opened it, a major version bump of Express, and it was just sitting there. The answer turned out to have almost nothing to do with Express, and by the end I'd touched a concurrency bug in a dependency, a broken release pipeline, and the fact that a webhook receiver already running in the cluster hadn't actually been receiving any of the fixes I thought it had.

The six-hour CI hang

The PR's check had failed after running for six hours, the default job cap on GitHub Actions, cancelled rather than failed outright. A test failure is normally fast; six hours means something hung and nothing was watching for it.

The fix for that part was mechanical: a per-test timeout so a stuck test fails in seconds with a clear error instead of silently eating the job's entire budget, plus a job-level cap as a backstop. The very next run after that change caught a websocket test that kept timing out waiting for an event that never arrived, reproducing reliably in CI and never once locally. That mismatch, CI-only and reproducible there every time, is the tell for an environment-specific bug rather than ordinary flakiness.

The bug lived in a small file-watching dependency the package used to tail a log file. Its change handler stored the in-progress read's file descriptor on a shared field, and two file-change events landing close together, routine under Linux's inotify but rarer under macOS's file events, let the second event overwrite that field before the first read's callback finished. The first callback then closed the wrong file descriptor, that read failed, and the error handler swallowed it silently, dropping a line that was supposed to stream to a connected client. Small package, and a bug that had nothing to do with the Express bump that started this; it just happened to be the PR unlucky enough to hit it first.

Replacing that dependency with a small in-house version fixed it outright, verified over repeated runs both locally and in CI where it had failed almost every time before.

Closing the auto-merge test gap

With that fixed, there was still a policy question sitting underneath it: major version bumps were deliberately excluded from auto-merge, on the reasoning that a breaking change deserved a human look. Worth asking directly what a human would actually catch there that a good enough test couldn't.

The honest answer, for this package specifically, was one gap: every existing test built both the web server and the router being tested from the same installed version of Express, because there was only ever one installed. The package's actual job is being mounted into someone else's Express app, possibly running a different major version than the one bundled inside it, and nothing tested that mismatch at all. An npm alias made it possible to install a second major version of Express purely for testing, build a host app on it, and mount the package's router into that mismatched host. It passed. With that gap closed, the case for holding every major bump for a human review went away, and the policy changed: any dependency bump merges once its tests pass, full stop.

The release pipeline had stopped running

Express 5 landed on main. Nothing published. The release pipeline only ran when the package's version field itself changed by hand, and nobody had touched it: every fix from the last several days had merged and gone nowhere.

The fix was switching to an automated release tool that reads commit types (a fix versus a feature versus a breaking change) and handles the version bump, the publish, and the changelog itself, no human editing a number ever again. Getting there took two more bugs, both from tooling rather than anything in the package: the release tool's current version needed a newer Node than the job was running, and separately, its own built-in npm-publishing step failed its authentication check against a publishing method that the plain npm publish command, run directly, already handled correctly in the exact same job. Routing the actual publish through the command that already worked, instead of the tool's own broken path to the same place, closed it. One manual version bump shipped everything that had piled up, and every release since has been automatic.

Deploying to the cluster

All of that fixed how the package gets published. It said nothing about the app already running in the cluster that depends on it, a small webhook receiver and live log viewer, the same one Part 5 covers moving behind a Cloudflare Tunnel. Checking turned up two separate gaps. Its lockfile was still pinned to an older version, because the automated dependency bump for it hadn't run yet on its own weekly schedule. And even once that bump landed, nothing would rebuild or redeploy anything, because no such pipeline existed yet. The pod had been running whatever image was built by hand, once, and would keep running it indefinitely.

Building that pipeline meant deciding where it could actually run from. The private registry this cluster uses lives on LAN-only DNS, and the cluster's own API server isn't reachable from outside it either, so a GitHub-hosted runner was never going to reach either one. The fix was a small self-hosted runner living as a pod inside the cluster itself, the only thing with a legitimate path to both, with just enough permission granted to it to restart one specific deployment and nothing else. Wired to trigger once the app's own tests pass on its main branch, it builds the image, pushes it, and rolls out the new pod, the same three steps a person would otherwise do by hand every time.

Verifying the pipeline end to end

Bumping the dependency by hand rather than waiting on next week's schedule, the whole chain ran once, live: a commit landed, tests passed, the image built and pushed, and a new pod came up with a different image digest than the one before it. A curl against the public endpoint, the same one anyone on the internet can reach, landed in the running app's log file and streamed live to a browser watching the viewer: proof that the path from a dependency fix to a pod serving real traffic still works end to end.

Summary

None of this started as a project. It started as one open pull request that shouldn't have still been open, and pulling on it surfaced a test suite that couldn't fail fast, a bug hiding behind a policy meant to guard against a different, less likely one, a release pipeline that had silently stopped doing its one job, and a running application with no way to find out a fix existed at all. Each of those was invisible until something forced a look. The chain now has no missing link: a fix merges, gets tested, gets published, and runs in front of real traffic. The only manual step left anywhere in that path is writing the fix itself.