Inside Oracle’s 25‑Million‑Line C Code: Why Fixing One Bug Takes Weeks

Recently a Hacker News thread asked for the worst code ever seen, sparking discussion. One comment highlighted Oracle Database 12.2, which contains nearly 25 million lines of C code.

The sheer size makes any change terrifying: a single line cannot be altered without breaking thousands of existing tests. Generations of developers have added layers of tangled, low‑quality code.

Complex logic, memory management, and context switches are tied together by thousands of flags and mysterious macros that often require days of manual expansion to understand.

Developers may need to reason about dozens, even hundreds, of flags to predict behavior under different conditions.

The product survives only because of millions of automated tests.

Typical day for an Oracle DB developer:

Start working on a new bug.

Spend two weeks trying to understand about 20 interacting flags.

Add another flag for a special case, write a few lines of code to handle it.

Submit changes to a test cluster of 100‑200 servers that rebuild the database and run millions of distributed tests.

Go home, return next day to tackle another bug; tests may take 20‑30 hours.

After another day, review test results, often seeing around 100 failures (or up to 1,000 in bad cases), and investigate dozens of flags to pinpoint the issue.

Add more flags, resubmit, wait another 20‑30 hours.

Repeat this two‑week cycle until you discover the “mysterious incantation” of flag combinations.

Eventually the fix passes without failures.

Add over a hundred new tests to protect future developers from breaking the fix.

Submit the final test results for review, which can take two weeks to two months, then move on to the next bug.

After two weeks to two months, the code finally merges into the main branch.

Fixing a small new feature can take six months to a year; adding a new authentication method may require two years. The product itself is a miracle, and the author vows never to work for Oracle again.

Source: Open Source China