Fixing a Chromium Process Leak in a Rust Ingestion Service

A production ingestion system I build and operate had accumulated 75,582 zombie processes. The Rust scheduler was still running, browser requests could complete, and the container continued passing its health check.

That made this a lifecycle investigation. A service could finish its work and still leave something behind on every run.

The ingestion service launched Chromium for pages that needed browser rendering. Chromium creates several helper processes. Some became orphaned when their parents exited, and the container had no init process to collect their exit statuses.

I reproduced the defect locally with one successful browser request. Without an init process, it left four zombies whose parent was PID 1.

The application itself was PID 1 inside the container. It managed the browser children it launched directly, but it did not provide the general orphan-reaping behavior expected from an init process.

The local reproduction established that a request did not need to fail for the leak to occur. It did not establish a fixed four-process leak rate for production; the 75,582 figure was the accumulated production count.

The health check only established that the main process existed. It could keep passing while exited browser descendants accumulated underneath it.

I added Docker init to both ingestion configurations so orphaned descendants would have a process responsible for reaping them.

I also made browser cleanup explicit. The service now requests shutdown and waits for exit, with separate cleanup deadlines so a timed-out request does not prevent cleanup from running. If graceful shutdown hangs, cleanup escalates to forced termination and waits again.

Cancellation needed its own fix. Testing exposed a race where a temporary profile could be deleted before Chromium finished exiting. The browser could then recreate files in that directory. Cleanup now retains ownership of the browser, its protocol handler and its profile until the shutdown sequence finishes.

A 2,048 process/thread limit adds containment if a future defect causes another runaway condition. The deployment also allows enough time for the scheduler's shutdown window. I recreated the ingestion container to clear the existing accumulation and apply the new configuration.

Validation covered 27 unit tests and 26 lifecycle scenarios, including successful requests, failed launches, oversized responses, timeouts, cancellation and a deliberately frozen browser requiring forced termination.

The final Linux run left zero browser processes, zero zombies and zero temporary profiles. The lifecycle regression is now included in CI.

The confirmed result was a reproduced process leak, repaired cleanup paths and a clean final regression run. Recreating the container removed the existing production buildup.

The important change is that successful work, cancellation and failed shutdown now have tested cleanup behavior.