Why the Linux OOM Killer nukes your database first: A rogue background worker leaked memory, but your core PostgreSQL instance paid the price.

Every production systems engineer has seen this horror story: A developer deploys a new Python background worker or Node.js processing script on a shared application host. The script has a slow memory leak. Hours later, the Linux kernel triggers the Out-Of-Memory (OOM) Killer.

Instead of terminating the leaking script, the kernel violently SIGKILLs your primary PostgreSQL or MariaDB daemon.

Here is the exact kernel logic that causes this catastrophe:

1. The Badness Heuristic: When physical RAM and swap are exhausted, the Linux kernel computes an ``oom_score`` for every running process. The primary input to this score is raw Resident Set Size (RSS)—the total physical memory pages held by that process. 2. The Database Target: A healthy database server deliberately buffers gigabytes of data in memory (shared buffers, cache pages). Even though the database was behaving perfectly, its massive memory footprint gives it the highest ``oom_score`` on the host. 3. The Collateral Damage: The kernel reaps the database process to reclaim the largest block of memory instantaneously. Transactions abort, dirty pages are abandoned, and the database is forced into crash recovery on restart.

Preventing rogue processes from taking down mission-critical daemons requires **systemd cgroups v2 resource slicing**: • Memory Protection with MemoryLow & MemoryMin: Configure systemd service units with ``MemoryMin=`` and ``MemoryLow=``. This instructs the kernel memory reclaimer that the database process's memory pages are strictly protected and must never be swapped or pruned under memory pressure. • Adjusting oom_score_adj: Explicitly protect critical daemons by setting ``OOMScoreAdjust=-900`` in the database service unit file. Conversely, configure untrusted background workers with ``OOMScoreAdjust=800`` so the kernel reaps the offender first. • Hard Slice Enclosure: Isolate workloads into dedicated systemd slices (``system.slice`` vs ``app.slice``). Enforce strict ``MemoryMax=8G`` and ``MemoryHigh=7G`` on application workers, terminating or throttling the leaking container within its own slice before host memory is threatened.

Never leave your production database's survival to default kernel memory heuristics.

Harden your Linux systems and container runtimes with our 2-week Hardened Linux & Container Infrastructure Sprint on Contra: https://contra.com/s/Xbg7ALqN-hardened-linux-and-container-infrastructure

#Linux #Performance #Infrastructure #Database #Architecture #DevOps #SRE