Restoring a publishing platform after recurring 503 errors

A publishing platform I build and operate generated a new release, activated it, and kept reporting healthy services. Visitors nevertheless received HTTP 503 instead of the website.

Rolling back restored access temporarily. Automated publication could then bring the failure back. I needed to identify the condition that made a completed release unusable and remove it without weakening the public server's validation.

The platform uses Elixir to generate immutable releases containing rendered pages, a content manifest and an inventory of render results. A Rust service checks that metadata and its integrity hashes before serving the pages.

The reader imposed a 16 MiB limit on metadata files. The previous manifest was 16,617,318 bytes. Its replacement was 16,805,625 bytes: just 28,409 bytes beyond the limit.

That was enough to make the new release unloadable. The reported metadata error referred to the size restriction; it was not evidence that the generated JSON was malformed.

The existing health checks missed the failure because they checked service availability. A responding health endpoint did not establish that the active site's release could be loaded and served.

I replaced the manifest's size-limited loading path with incremental parsing and hashing. The reader now retains the routing information it needs and discards editorial page payloads as it parses them.

I kept the checks that protect the serving process: valid JSON, the exact-byte SHA-256 integrity seal, and bounds on tokens, nesting, route counts, inventory and serving-index memory. Removing the overall manifest-size ceiling did not mean accepting unlimited resource use.

This addressed the recurring failure at its source. Raising the old ceiling would have left the same growth-dependent failure waiting at a larger number.

The incident also exposed a separate browser cleanup problem in the Go and Lighthouse sidecars. Process inspection found 9,191 zombie children across the two deployment slots.

I corrected cleanup after cancellation, connection failures and incomplete browser shutdown, and enabled container init to reap orphaned descendants. After verifying cutover, deployment also rebuilt and recreated the former slot's public reader and browser sidecars. That removed the old process buildup and prevented the retained slot from keeping the incompatible reader.

A cancelled PostgreSQL lock query was investigated separately. The evidence did not establish a deadlock, and this patch did not rewrite database locking.

A production Rust container served a 163.9 MB manifest containing 5,000 page payloads. The same reader still rejected an invalid integrity seal.

The separate process reproduction produced 160 zombies without init and zero with init. Container recreation also cleared a test environment containing both zombies and live orphan processes.

Broader application tests and production image builds supported the release. After deployment, I confirmed that the public site was serving again and the observed zombie buildup was gone.

The result was a recovered website and a reader tested beyond the old failure boundary, with its integrity checks and resource bounds preserved.