42 Lessons Learned from Building a Production Database – Translated Summary

Customer (User)

(1) Keep your customers happy; otherwise the rest of the article is irrelevant.

(2) Start with the right number of customers (initially just one) and the right kind of customers who allow you to build critical technology, and increase this number carefully.

(3) Communicate directly with customers; many internal conflicts can be resolved by saying, "I just talked to the customer, they said...".

(4) Recognize that customers may not be able to articulate their true needs; spend time understanding their use‑cases and read their code.

Project Management

(5) Have a clear mission statement; Delos’s mission is to become a reliable foundation for Facebook infra.

(6) Re‑evaluate task difficulty repeatedly; decision‑makers often lack time, context, or training and may mis‑estimate by orders of magnitude.

(7) Assign tasks to ICs (Individual Contributors) strategically; stay on the critical path because you usually understand the problem better than managers.

(8) Treat the roadmap as a means, not an end.

(9) If you have a good manager, try to understand, support, and tolerate them; if not, seek clarification.

(10) Build projects robust to organizational changes; ensure manager turnover does not unfairly affect ICs.

(11) Track how long similar features took in other projects to gauge task difficulty.

Design

(12) Be conservative on APIs and liberal with implementations.

(13) Introduce new implementations cautiously (gradual, staged rollout).

(14) When designing an API, implement the first version, actively plan a second, and hope a third will work.

(15) Design APIs with migration to new implementations in mind; avoid custom migrations.

(16) Design as a team, implement as individuals.

(17) For storage systems, prioritize consistency and durability over availability early on.

(18) Maintain multiple implementations in tests and compare results.

(19) Use late binding in design to keep the design space open.

(20) Allow any IC to implement the design after it is finalized.

(21) Have the right amount of abstraction – not too few (monolith) nor too many (cognitive overload).

(22) Avoid using real‑time guarantees for correctness unless clock error bounds are understood.

(23) Maintain a single source of truth and simple invariants across states.

(24) Foster a culture where ICs constantly explore alternative designs.

(25) Understand your SKU; hardware knowledge remains crucial despite cloud abstraction.

Code Review

(26) In fast‑review, transparent repositories, guard APIs against leaking implementation details.

(27) Encourage critical thinking on diffs and create an environment where feedback is welcomed.

(28) For critical components, require multiple approvals (e.g., two LGTM).

(29) Do not equate landing time with importance; allow longer review cycles for high‑quality work.

(30) Resist the urge to land code before confirming the design is correct; allow discarding code if it’s not the right solution.

Strategy

(31) Periodically ask why the team/project exists and how it adds value.

(32) Track other major projects in the same domain and be able to explain their designs better than their own ICs.

(33) Avoid competing on raw performance; focus on fundamental design characteristics.

(34) If another system better fits the use‑case, step aside.

Observability

(35) Measurement is a means, not an end.

(36) Detect service issues before customers do.

(37) Place observability above APIs and outside implementations to enable easy switching and comparison.

(38) Pay special attention to hard‑to‑measure properties like consistency.

(39) Push critical checks (e.g., consistency) into deployments, minimizing external service checks.

Research

(40) Track research results in your field to quickly combine ideas across projects.

(41) Experiment with new approaches; resist copying designs verbatim.

(42) Write papers to clarify assumptions, attract talent, and deepen understanding.