Can Modern OSes Replace Database Kernels? Insights on OS‑Database Synergy

The author reflects on a recent discussion about delegating more responsibilities to the operating system (OS) so that database kernels can be simplified. This idea recalls the long‑standing debate over the O_DIRECT file‑I/O option, where Linus Torvalds famously warned that bypassing the OS is fundamentally broken and only an “idiot who writes databases” would consider it.

Historical Dependence on the OS

Early databases were tightly coupled with the OS. For example, the RMS record‑management system ran on OpenVMS and relied on the OS’s inter‑process communication primitives. As databases grew more complex and needed to run on many platforms, they began to re‑implement functions that the OS previously provided.

A 2012 benchmark on an Oracle T4‑8 server running Solaris with Oracle 11g demonstrated the performance gains possible when the OS and database cooperate closely. Minimal tuning and a few database‑parameter tweaks yielded outstanding results, largely because the OS optimized lock and latch handling for Oracle’s concurrency control.

Modern OS Capabilities

Today’s Linux has evolved far beyond the early versions that could only support simple web workloads. Contemporary databases such as Redis, MongoDB, ClickHouse, MySQL, and PostgreSQL all exploit advanced OS features: file‑cache buffering, log‑structured file systems, and other kernel‑level optimizations that reduce the need for custom database code.

When OS Optimizations Aren’t Enough

In high‑load, specialized scenarios the OS’s automatic optimizations may fall short. The 2007 O_DIRECT controversy highlighted cases where database vendors needed direct control over I/O to achieve required performance, bypassing the OS cache.

From a DBA’s perspective, managing the database’s own buffer pool can be more efficient than relying solely on the OS file cache. The database can make nuanced decisions about which pages to evict, based on hot‑data heuristics that the generic OS cache cannot replicate.

Practical Recommendations

For most workloads, the OS file cache provides a solid performance boost. PostgreSQL’s documentation advises allocating only 20‑30 % of memory to the shared buffer pool and leaving the rest to the OS. Some users, however, prefer to give the database a larger share of RAM, arguing that a bigger shared buffer yields better performance for their specific workloads.

Optimization Layers

System optimization follows a bottom‑up hierarchy: low‑level OS tuning is cheap and easy, while higher‑level application‑specific optimizations demand more expertise and investment. Small, simple applications often benefit sufficiently from OS‑level tweaks, whereas large, complex systems may need manual tuning akin to choosing a manual transmission over an automatic one for demanding terrain.

Future Outlook

As operating systems continue to improve, more responsibilities can shift back to the OS, reducing the need for custom database code. Certain specialized databases already benefit from OS‑level enhancements, and cloud‑native or managed database services (e.g., RDS) can leverage OS upgrades to gain performance without changing the database software itself. Custom OS tuning remains a viable strategy for workloads that demand the utmost efficiency.