Every Deploy Said Success. The Endpoint Was 404ing.

An internal endpoint on the platform I work on had been returning 404 for about two and a half weeks. Nobody had filed a ticket, every deployment in that window had reported success, and the service behind it was healthy the whole time — running, warm, serving.

I only found it because I was auditing an architecture diagram against the live system, expecting to correct the diagram.

The moment that turned this from confusion into a diagnosis was noticing that two different paths 404'd differently — one returned text/html, the other text/plain.

text/html — the request never reached the service at all. The hosting layer has a /** catch-all that serves the frontend single-page app, and the SPA returned its own not-found page. No rewrite existed for this path.

text/plain — the request did reach the service. This is the Go router 404ing internally, because that exact path is not registered on it. The rewrite exists and points at a live service, but the service disagrees about the path.

application/json — the service is running and this is a real application response. Once I had the route fixed, hitting it returned 400 application/json from the controller's own field validation, which is the sound of success.

So the two 404s were telling me two halves of the same story: the old path had a rewrite that no longer matched anything the service served, and the new path had no rewrite at all. Somewhere, a rename had happened on one side of a boundary and not the other.

A request to the public API domain hits a hosting layer that holds a list of rewrites — path to Cloud Run service. That list is generated by a deploy script and stored in a secrets manager. Separately, the service itself registers its routes in its Go router. Both must agree, and nothing enforces that they do.

The git history is unambiguous. On 19 June the two were in sync. On 20 July a single commit renamed the route in the Go file and left the rewrite untouched. On 6 August I found it during an unrelated diagram audit.

One commit, one side of the boundary. The commit was reviewed. It was also entirely reasonable in isolation — it renamed a route in a Go file, and there is nothing in that Go file that hints a copy of the path is stored in a secrets manager.

This is the part I find genuinely interesting, and the reason the bug survived seventeen days of active development.

The deploy script does not rewrite the whole route table. It merges the entry for the service being deployed into the existing table, keyed by service id. Every entry that is not the service being deployed is carried forward verbatim. That is a sensible design — it means deploying one service cannot clobber another's route.

It also means a wrong entry is immortal. It will be faithfully preserved by every future deploy of every other service, forever, until somebody redeploys that one specific service's hosting config.

The version history bore this out exactly. Across ten consecutive versions of the route table, the broken entry is byte-identical in every one — including two versions written by my own deploys of an unrelated service earlier that week. My deploys succeeded. They also carefully preserved the bug.

The deploy script is not naive. It has a guard that refuses to deploy if the /** catch-all is not the last rule, because a catch-all above an API route shadows it. That guard is correct, and it is well-reasoned. It also has a smoke test that does exactly the content-type check described above. Three things are wrong with it, and they are all instructive:

It probes the wrong host. The default hosting domain is not the custom domain that real traffic uses. Those can resolve to different hosting configurations.

It warns rather than fails. A warning in a CI log that already printed "deployed successfully" is not a signal anybody reads.

It only checks the route being deployed right now. This is the fatal one. The broken route belonged to a service nobody was deploying. A check scoped to the change can never catch a regression in something you did not change.

That third point is the general lesson. Most deploy-time validation asks "did my change land?" Almost none asks "is everything still reachable?" — and the second question is the one that catches drift.

One more, because it cost me an hour separately and it is not obvious: a /** rewrite does not match its own bare parent. A rewrite on /api/projects/v1/detail/** will match /api/projects/v1/detail/123 but not /api/projects/v1/detail, which falls through to the frontend catch-all. If your API has an endpoint that is both a collection and a prefix, you need both rules. I verified this against a live deployment rather than trusting the docs, and I would suggest you do the same, because the behaviour differs between hosting providers.

Derive, don't duplicate. The real fix is for the route to exist in exactly one place. The service already knows its own path — it is in an environment variable the service reads at startup. A deploy that asked the service for its route, instead of being told the route separately, could not drift.

Check the whole surface, not the delta. A job that walks every registered route and asserts each returns something other than text/html from the public domain would have caught this on day one. It is a loop over a list you already have.

Fail, don't warn. If an assertion is worth writing, it is worth breaking the build over. A warning printed after "deployed successfully" is decoration.

Read the whole response, not the status code. A 404 with text/html, one with text/plain and one with application/json are three different bugs in three different places. The status code told me nothing; the content-type told me exactly which layer to open.

"Merge" is a synonym for "carry forward the mistakes." Any config assembled by merging into previous state will preserve errors indefinitely, and every subsequent successful deploy becomes evidence that things are fine. If you merge config, you need a periodic check of the merged result — not of the merge.

Validation scoped to the change has a blind spot exactly the size of everything else. This bug lived in the gap between "my deploy worked" and "the system works."

Silence is not health. Seventeen days, no ticket. The endpoint was called by one internal flow that people had stopped exercising. Absence of complaints measured attention, not correctness.