DBOS: A Database‑Oriented Operating System for Cloud Computing

In 2021, Matei Zaharia, the founder of Databricks and creator of Spark, described at Stanford how the company struggled to schedule and track millions of Spark tasks using PostgreSQL, which proved too slow and unsuitable for distributed, lightweight task management.

Databricks chose PostgreSQL for task state storage, but the database could not handle the rapid influx of status updates for a million tasks, leading to performance complaints that even impressed database pioneer Michael Stonebraker.

The article then surveys historical attempts to embed databases in operating systems, such as IBM’s AS/400 and Microsoft’s WinFS, showing that storing files and system state in relational tables is not new.

Building on these ideas, Zaharia and Stonebraker propose DBOS (Database‑Oriented Operating System), a layered architecture where a microkernel runs a high‑performance, multi‑node distributed database, and OS services like scheduling, messaging, and a distributed file system are implemented on top of the DBMS.

The architecture consists of four layers: (1) microkernel for memory and drivers, (2) a distributed transactional database, (3) OS services built on the DBMS, and (4) user applications such as Spark.

DBOS offers several benefits: task state is stored in a database, enabling ACID‑guaranteed concurrency, easy recovery via database roll‑back, comprehensive logging for debugging, and a reduced attack surface compared to traditional OS stacks.

As an illustration, DBOS can run reliable long‑running workflows; the following code shows a simple reminder workflow that persists across crashes and restarts:

@DBOS.workflow()
def reminder_workflow(email: str, time_to_sleep: int):
    send_confirmation_email(email)
    DBOS.sleep(time_to_sleep)
    send_reminder_email(email)

The article concludes that DBOS is not intended as a desktop OS but as a cloud‑native operating system that could reshape how distributed applications are built, though it remains a proprietary project unlike open‑source PostgreSQL.