Why ‘Suffering‑Oriented Programming’ Can Reduce Risk in Large‑Scale Projects

Nathan Marz, the primary author of Twitter's distributed stream‑processing framework Storm, released his new book "Building Data‑Intensive Applications"; this is a Chinese translation of his 2012 blog post sharing his development philosophy.

Suffering‑Oriented Programming

When asked how he could dare to write Storm at a startup, Marz explains that large‑scale projects are not inherently risky if approached with a development style he calls “Suffering‑Oriented Programming.” This approach means only working on projects that cause real pain when missing, thereby focusing effort on truly important problems.

The three maxims of this philosophy are: “First make it possible, then make it beautiful, finally make it fast.”

First Make It Possible

When entering an unfamiliar problem domain, avoid building a generic or extensible solution before understanding the real requirements; instead, quickly hack a solution to address the immediate need. In Storm’s early stage, a year‑long rapid development built a streaming system using queues and worker nodes, learning ack protocols, partitioning strategies, and real‑time scaling.

Initially, the focus was on product functionality rather than fault tolerance; the queue‑worker paradigm proved useful despite its complexity, and the need for distributed processing became evident when calculating the reach of a URL on Twitter required hundreds of database accesses and massive deduplication.

Then Make It Beautiful

After hacking a solution, the “map” of the problem domain becomes clearer, allowing the design of elegant abstractions that solve known concrete problems without over‑engineering for unknown cases. This stage involves extracting the simplest set of abstractions, akin to fitting a regression line to data points.

In Storm, the real‑time processing model was distilled into streams, spouts, bolts, and topologies, eliminating intermediate message proxies while ensuring data processing, thereby unifying seemingly disparate problems.

Finally Make It Fast

With a beautiful design in place, safe performance profiling and optimization become possible; premature optimization is avoided because the design may still change. “Fast” refers to micro‑level code efficiency—reducing constant factors and tightening resource usage—rather than high‑level system performance.

Iterative Refinement

Suffering‑Oriented Programming is a continuous process: each elegant system grants new capabilities, leading to deeper domain understanding, which feeds back into further refinements. Storm evolved through multiple iterations, adding features such as direct streams, transactional topologies, and handling at‑least‑once semantics.

The approach embraces refactoring to combat accidental complexity, emphasizing that code should evolve based on real use cases rather than speculative generality.

Conclusion

Use cases are the gold of Suffering‑Oriented Programming; the only way to acquire them is by writing code and gaining experience. Programmers typically progress from struggling to get things running, to recognizing the benefits of structured and shared code, to abstracting and eventually generalizing their solutions, always driven by concrete, painful problems.