Health checks that paid for themselves during a release

Early in my time on the project I built health checks for a set of legacy .NET services running on IIS: not endpoints that always answer 200, but checks that actually open a database connection and roll up into an aggregated status. A few weeks later they turned out to be the most useful thing I had built.

The release

The quarterly release bundled roughly 600 hours of work. The service window ran in the evening for two hours, with a fixed time by which we had to decide whether to roll back. That morning the build pipeline was broken: NuGet restore failed because a local package source in NuGet.config resolved to a path that did not exist on the build agent. With that fixed, I staged the new build next to the live sites, backed up the old services and updated connection strings by hand on each server.

The incident

After the cutover, several services could not reach their databases. A typo in a connection string had been copied to more than one server. Instead of treating the whole environment as down, the health checks showed exactly which services on which servers were failing, and the fix took 20–30 minutes, well inside the window.

What I took from it

• A health check is only worth having if it tests the real dependency.
• Hand-edited configuration across servers is where releases break. That is a large part of why the new platform’s pipelines own configuration per environment instead.