Why a Database‑First Operating System Could Replace Linux and Kubernetes

Problem Statement

Databricks runs millions of tiny Spark tasks across a cluster. The original design stored task state (queued, running, succeeded, failed) in PostgreSQL. While PostgreSQL is reliable, it cannot sustain the write‑throughput required to record rapid state changes for a million‑scale workload, leading to latency and scalability bottlenecks.

Historical Context

Embedding a relational database in the operating environment is not new. IBM’s AS/400 stored files as rows in a database table, and Microsoft’s WinFS project attempted to manage files, contacts, and email metadata via a relational store. Both demonstrated the idea of a database‑backed OS but remained tied to a traditional OS stack.

DBOS Architecture

DBOS (Database‑Oriented Operating System) inverts the classic stack: a high‑performance, multi‑node distributed database sits **below** a microkernel, and OS services are built **on top** of the DBMS.

Layer 1 – Microkernel : Handles memory management, interrupt handling, and device drivers.

Layer 2 – Distributed Database : Provides transactional, highly available storage across many nodes.

Layer 3 – OS Services : Implements a distributed file system, cluster scheduler, and inter‑process communication (IPC) as database‑backed services.

Layer 4 – User Applications : Runs workloads such as Spark, Spark‑SQL, or custom services.

All OS and application state is represented as rows in the underlying database and accessed exclusively through ACID transactions.

Key Properties and Benefits

Unified State Store : Task metadata, file system metadata, and application data share the same tables, making massive state tracking a simple CRUD operation.

Transactional Guarantees : Concurrent updates are serialized by the database engine, ensuring consistency and durability even after node failures.

Built‑in Auditing : Database write‑ahead logs provide fine‑grained change histories for debugging, monitoring, and security analysis.

Instant Rollback : A snapshot or point‑in‑time restore of the database reverts the entire OS to a prior state, reducing attack surface and simplifying recovery.

Workflow Example

DBOS includes a workflow runtime that persists the progress of long‑running processes. The following Python‑style example demonstrates a reminder workflow that sends a confirmation email, sleeps for a configurable interval, then sends a reminder email. The workflow state is stored in the database, so it survives crashes or restarts.

@DBOS.workflow()
def reminder_workflow(email: str, time_to_sleep: int):
    send_confirmation_email(email)
    DBOS.sleep(time_to_sleep)
    send_reminder_email(email)

Because the workflow is executed inside a transaction, DBOS can guarantee exactly‑once delivery of the confirmation email and eventual delivery of the reminder, regardless of how long the sleep period lasts.

Limitations and Outlook

DBOS is not intended as a general‑purpose desktop OS like Linux or Windows. Its design targets cloud‑native, distributed workloads where a unified, transactional state store can replace layers such as Kubernetes, external schedulers, and separate metadata services. Adoption depends on the availability of a production‑grade distributed database and the willingness of cloud providers to run workloads on a database‑centric OS.

References

https://www.youtube.com/watch?v=CDSgJE5mPJM

https://thenewstack.io/meet-dbos-a-database-alternative-to-kubernetes/

https://www.nextplatform.com/2024/03/12/the-cloud-outgrows-linux-and-sparks-a-new-operating-system/

https://venturebeat.com/data-infrastructure/postgresql-and-databricks-founders-join-forces-for-dbos-to-create-a-new-type-of-operating-system/

https://www.infoworld.com/article/2336467/dbos-cloud-overturns-database-on-os-conventions-for-speed.html

https://dbos-project.github.io/blog/intro-blog.html