42 Lessons Learned from Building a Production Database – Translation

Recently I read Mahesh Balakrishnan’s blog "42 Things I Learned from Building a Production Database" and translated the entire text, as the insights are highly valuable for anyone working on infrastructure.

Customer (User)

Make your customers happy; everything else is irrelevant.

Start with a single, well‑chosen customer who allows you to build core technology, and grow the number carefully.

Talk directly to customers to avoid internal speculation.

Understand customers’ real needs by reading their code, not just their stated requirements.

Project Management

Define a clear mission statement (e.g., "We will become FB infra’s reliable foundation").

Continuously reassess task difficulty; decision‑makers often mis‑estimate.

ICs should stay on the critical path of decisions because they know the code and strengths best.

A roadmap is a means, not an end.

Support good managers and understand their constraints; if none exist, seek alternatives.

Ensure project robustness against organizational changes; protect ICs from unfair career impacts when managers rotate.

Track how long similar features took in other projects to benchmark effort.

Design

Be conservative with APIs and liberal with implementations.

Introduce new implementations gradually (canary, phased rollout).

When designing APIs, implement the first version, plan the second, and hope the third works.

Design for easy migration to new implementations; provide a single CLI switch.

Design as a team, implement as individuals to avoid parallel design chaos.

For storage systems, prioritize consistency and durability over availability early on.

Maintain multiple implementations in tests and compare results.

Use late binding to keep the design space open and encourage prototyping.

Allocate tasks to ICs based on their expertise; keep decision‑makers on the critical path.

Avoid over‑optimizing raw performance; compete on fundamental design features instead.

Observability

Measurement is a means, not a goal.

Detect problems before customers do.

Place observability above APIs and outside implementations to enable easy swapping.

Pay special attention to hard‑to‑measure properties like consistency.

Push critical checks (e.g., consistency) into deployment, minimizing external service checks.

Research

Track research in your domain to quickly combine ideas from different projects.

Experiment with new approaches; resist copying designs verbatim.

Write papers to clarify assumptions and attract talent; present them when possible.

Strategy

Regularly ask why the team exists and how it adds value to the company.

Know other major projects in the same area and be able to explain their design choices.

Avoid competing on raw performance; focus on core design trade‑offs.

If a better system exists for your use‑case, consider adopting it instead of reinventing.

The article concludes with an invitation for readers to discuss these points, ask questions, and share their own experiences.