The Inevitability of Complexity in Software Design

The struggle with complexity is an eternal theme in software development, repeatedly surfacing at every level with debates about how much commenting a function should have, what ideal abstraction looks like, when a framework becomes overly magical, and when an organization has too many languages.

We try to escape, control, and simplify complexity, but seeking solutions solely through simplicity is misguided because complexity must exist somewhere and is indestructible.

Resilience Engineering teaches a cybernetic concept of "necessary diversity": only complexity can handle complexity.

When dealing with build tools, several truths become evident:

If you make the build tool simple, it cannot handle all the strange edge cases that exist.

If you want to handle strange edge cases, you must deviate from any norms you wish to establish.

If you want generic defaults to be easy to use, the rules for those defaults must be shared between the tool and the user, and the user must adapt their system to the tool.

If you allow self‑configuration or scripting, you give users a way to specify shared rules so the tool fits their system.

If you want to keep the tool simple, you must force users to operate only within parameters that suit that simplicity.

If a user's use case does not match the tool's simplicity, they will create shims around the tool to achieve their goals.

This is unavoidable; complexity must exist somewhere, whether we are aware of it or not, and it is always a factor people need to consider when solving problems.

Unfortunately, when we reach the last point of the list—inevitably creating shims—the complexity becomes part of the overall tool and does not disappear. This portion of complexity is something everyone must learn, and users will need to adapt to it.

People trying to bypass complexity encounter a mismatch between two conflicting concepts (similar to the O and R concepts in ORM). This necessary complexity may shift back to the tool (or a new tool) or be eliminated by rebuilding things. Each such change requires more effort and adjustment, making people see, understand, and address complexity. However, sometimes these changes do not simplify matters but create new mismatches among assumptions, leading to further complexity and new shims. Over time, necessary complexity becomes accidental complexity.

In Don Norman's "The Design of Everyday Things," the concepts of "knowledge in the head" and "knowledge in the world" are introduced (similar ideas appear academically in Roesler & Woods' "Designing for Expertise"). Knowledge in the head is what you know, have learned, and remember. Knowledge in the world comprises everything else: written information, design cues (e.g., recognizing a power button because it looks like a button). Interpreting world knowledge involves cultural and contextual factors and relies on head knowledge.

In some respects, expertise is the knowledge in your head that enables you to better understand the world.

Focusing on how "simple" a piece of code is to read and interpret is a common trap in software design. Emphasizing simplicity is dangerous because complexity cannot be eliminated; it can only be transferred. If you move it out of your code, where does it go?

When we designed Rebar3, we thought the tool could be simple, assuming users had a basic understanding of the expected structure of an Erlang/OTP project. Following those rules makes projects run well, but we externalized some complexity to the broader ecosystem. Those rules always need to be learned, and the tool now depends on understanding them. Simplifying the tool for those who know the rules makes it harder for those still learning them.

In software architecture this trap is subtle. When we adopt micro‑services or similar approaches, we try to make each service simple. However, unless you can force your real application into a simple state, complexity still has to reside somewhere. If it is not in each micro‑service, where does it go?

Complexity must exist somewhere. If you are lucky, it resides in well‑defined places: in slightly complex code, supporting documentation, or engineering training courses. You give complexity a place instead of trying to hide it, creating methods to manage it and knowing where to look when needed. If you are unlucky and pretend complexity can be fully removed, it has nowhere to go and will still persist in the world.

Because it has nowhere to go, complexity wanders throughout your system, including in your code and people's minds. As people move on, our understanding of it gradually weakens.

Complexity must exist somewhere and will not disappear. If you accept it, give it its proper place, recognize its existence when designing systems and organizations, and focus on adapting to it, it can become a force that helps software architecture design.

Author: Yao Gangqiang Source: https://docs.google.com/document/d/1Qp2foEIk2Tn0x-hwM3-3FeFf3sADnhMabQGOkgE09QI Copyright statement: Content sourced from the internet for sharing and learning purposes; copyright belongs to the original author. We will indicate the author and source unless unverified, and will remove any infringing material upon notice.

Architect's Guide

We are all architects!