42 Lessons Learned from Building a Production Database

Customer (User)

(1) Keep your customers happy; otherwise the rest of the article is irrelevant.

(2) Maintain the right number of customers (start with one) and the right customers who allow you to build key technology, and increase this number carefully.

(3) Engage directly with customers; many internal conflicts can be resolved by saying, "I just talked to the customer, they said...".

(4) Recognize that customers may not be able to articulate their true needs; read their code and understand their use cases deeply.

Project Management

(5) Have a clear mission statement; Delos’s mission is to become the reliable foundation for FB infra.

(6) Repeatedly assess task difficulty; decision‑makers may lack time or context, leading to large estimation errors.

(7) Task allocation to ICs is critical; stay on the critical path of decisions because you often know the problem and codebase better than managers.

(8) Road‑maps are a means, not an end.

(9) If you have a good manager, understand and support them; if not, seek clarity.

(10) Make projects robust to org‑chart changes; communicate constantly with future managers to avoid unfair career impacts for ICs.

Code Review

(26) In a fast‑review, transparent codebase, APIs can leak implementation details unless guarded.

(27) Encourage ICs to critically examine diffs and foster an environment where feedback is welcomed.

(28) For critical components, consider informal rules like requiring two LGTM approvals.

(29) Do not equate delivery speed with importance; allow time for thorough reviews of high‑quality work.

(30) Resist the urge to ship first and fix later; allow ICs to discard code that isn’t the right solution.

Design

(12) Be conservative on APIs and liberal with implementations.

(13) Introduce new implementations cautiously (gradual roll‑out).

(14) When designing APIs, implement the first version, actively plan the second, and hope the third works.

(15) Design APIs with migration to new implementations in mind; avoid custom migrations.

(16) Design as a team, implement as individuals.

(17) For storage systems, prioritize consistency and durability over availability early on.

(18) Maintain multiple API implementations in tests and compare results.

(19) Use late binding to keep design space open and prototype key paths.

(20) After design, any IC should be able to write the code.

(21) Have the right amount of abstraction—neither too few nor too many.

(22) Avoid using real‑time guarantees for correctness unless clock error bounds are understood.

(23) Maintain a single source of truth and simple invariants across state types.

(24) Cultivate a culture where ICs constantly explore alternative designs.

(25) Understand your SKU; hardware knowledge is crucial for design.

Observability

(35) Measurement is a means, not an end.

(36) Detect service problems before customers do.

(37) Keep observability above the API and outside implementations to enable fair performance comparisons.

(38) Pay special attention to hard‑to‑measure properties like consistency.

(39) Push critical checks (e.g., consistency) into deployment and minimize external service checks.

Research

(40) Track research in your domain to quickly prototype and combine ideas across projects.

(41) Favor new approaches over copying existing designs; breakthroughs often start as half‑baked ideas.

(42) Write papers to clarify assumptions, attract talent, and deepen understanding; encourage talks and lectures.

Source: 多颗糖 Link: https://maheshba.bitbucket.io/blog/2021/10/19/42Things.html Copyright notice: Content sourced from the internet; credit to the original author.