𝗪𝗲 𝗿𝘂𝗻 𝗮𝗻 𝗘𝗱𝗧𝗲𝗰𝗵 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝘁𝗵𝗮𝘁 𝗴𝗿𝗲𝘄 𝗳𝗿𝗼𝗺 𝟮𝟬,𝟬𝟬𝟬 𝘁𝗼 𝟮𝟱𝟬,𝟬𝟬𝟬 𝗱𝗮𝗶𝗹𝘆 𝘂𝘀𝗲𝗿𝘀 𝗶𝗻 𝘁𝗵𝗿𝗲𝗲 𝘆𝗲𝗮𝗿𝘀, 𝘄𝗶𝘁𝗵 𝘇𝗲𝗿𝗼 𝗱𝗼𝘄𝗻𝘁𝗶𝗺𝗲.

The hardest part wasn't the growth. It was Monday at 8 AM, when 20,000 students logged in within 90 seconds and the whole system buckled.

𝗪𝗵𝗮𝘁 𝗳𝗶𝘅𝗲𝗱 𝗶𝘁:

• Login got its own service, so it never touched the main database. The Monday slowdown became a 3-second blip. • Submissions went through a queue. The slowest endpoint got 80% faster. • Reporting got its own database. Teacher dashboards went from 14 seconds to under 1.

𝗧𝗵𝗲 𝗹𝗲𝘀𝘀𝗼𝗻: going from 50K to 150K users breaks more platforms than going from 150K to 250K. The first jump is an architecture problem, not an engineering one.

Full case study on my profile. If your product is starting to strain under real users, send me what you're building.

Why the Linux OOM Killer nukes your database first: A rogue background worker leaked memory, but your core PostgreSQL instance paid the price.

Every production systems engineer has seen this horror story: A developer deploys a new Python background worker or Node.js processing script on a shared application host. The script has a slow memory leak. Hours later, the Linux kernel triggers the Out-Of-Memory (OOM) Killer.

Instead of terminating the leaking script, the kernel violently SIGKILLs your primary PostgreSQL or MariaDB daemon.

Here is the exact kernel logic that causes this catastrophe:

1. The Badness Heuristic: When physical RAM and swap are exhausted, the Linux kernel computes an ``oom_score`` for every running process. The primary input to this score is raw Resident Set Size (RSS)—the total physical memory pages held by that process. 2. The Database Target: A healthy database server deliberately buffers gigabytes of data in memory (shared buffers, cache pages). Even though the database was behaving perfectly, its massive memory footprint gives it the highest ``oom_score`` on the host. 3. The Collateral Damage: The kernel reaps the database process to reclaim the largest block of memory instantaneously. Transactions abort, dirty pages are abandoned, and the database is forced into crash recovery on restart.

Preventing rogue processes from taking down mission-critical daemons requires **systemd cgroups v2 resource slicing**: • Memory Protection with MemoryLow & MemoryMin: Configure systemd service units with ``MemoryMin=`` and ``MemoryLow=``. This instructs the kernel memory reclaimer that the database process's memory pages are strictly protected and must never be swapped or pruned under memory pressure. • Adjusting oom_score_adj: Explicitly protect critical daemons by setting ``OOMScoreAdjust=-900`` in the database service unit file. Conversely, configure untrusted background workers with ``OOMScoreAdjust=800`` so the kernel reaps the offender first. • Hard Slice Enclosure: Isolate workloads into dedicated systemd slices (``system.slice`` vs ``app.slice``). Enforce strict ``MemoryMax=8G`` and ``MemoryHigh=7G`` on application workers, terminating or throttling the leaking container within its own slice before host memory is threatened.

Never leave your production database's survival to default kernel memory heuristics.

Harden your Linux systems and container runtimes with our 2-week Hardened Linux & Container Infrastructure Sprint on Contra: https://contra.com/s/Xbg7ALqN-hardened-linux-and-container-infrastructure

#Linux #Performance #Infrastructure #Database #Architecture #DevOps #SRE