Postgres Stability: The Power of Strict Memory Overcommit

Decoding Postgres Memory Overcommit

The article provides a compelling argument for strict memory overcommit in PostgreSQL, effectively demonstrating the catastrophic failure modes of the default Linux kernel policy. The detailed explanation of how a SIGKILL event forces a full instance restart, including WAL replay, is crucial for understanding the real-world impact. The comparison with strict overcommit, where failures are localized to individual queries and transactions, highlights a significant improvement in resilience. The practical demonstration on an EC2 instance, showing the stark difference in failure scenarios and downtime, lends strong credibility to the findings. The calculation for the vm.overcommit_kbytes limit, factoring in huge pages, kernel overhead, and per-backend allocations, is technically sound and demonstrates a deep understanding of both Linux memory management and PostgreSQL's resource consumption patterns. The inclusion of headroom for sidecar processes further refines this practical approach.

However, while the article strongly advocates for strict overcommit, it primarily focuses on its benefits in preventing catastrophic failures. A potential limitation is the exploration of scenarios where the vm.overcommit_kbytes limit might be set too conservatively, leading to legitimate queries failing prematurely even when ample physical memory is available. While the article mentions this as a concern and details the calculation to avoid it, a more in-depth discussion or case study on tuning this limit in diverse workloads could further enhance its practical value. Additionally, while the performance difference is shown to be negligible, some users might still have concerns about any potential overhead associated with the kernel's stricter accounting, even if not discernible in the benchmark. Future discussions could touch upon advanced monitoring techniques to identify and address such edge cases proactively.

Key Points

• Linux kernel's default memory overcommit policy can lead to catastrophic PostgreSQL instance restarts (SIGKILL) when physical memory is exhausted, causing significant downtime and data loss.
• Strict memory overcommit (vm.overcommit_memory = 2) prevents this by enforcing a commit limit, causing individual queries to fail with ENOMEM instead of killing the entire backend process.
• This localized failure allows transactions to roll back gracefully, keeping other connections active and preventing instance-wide outages.
• ClickHouse Managed Postgres implements strict overcommit with a carefully calculated commit limit (roughly 60% of total RAM + 2GB) to balance safety and resource utilization.
• The performance overhead of strict overcommit is negligible and indistinguishable from noise in benchmarks.

---

📖 Source: Why strict memory overcommit matters for Postgres