Applying Layered Architecture to a Massive Python Backend Project

TLDR (AI Claude summary): A Python developer describes the layered architecture used in a massive Python project of about 30,000 files maintained by over 400 developers, detailing how the structure is enforced with Import Linter and its impact on complexity and technical debt.

Introduction – The author, a Python developer at Kraken Technologies, maintains a Python application with roughly 27,637 modules (nearly 28 K files) used by 17 energy and utility companies. Despite its size, the project remains maintainable thanks to a disciplined code‑organization strategy.

Layered Architecture – The team adopted a layered (stack‑like) architecture where higher layers may depend on lower layers, but not vice‑versa. Components are defined as logical layers (e.g., client, territories, core) and dependencies are restricted to flow downward.

In the diagram, component C can depend on A and B but not on D, illustrating the downward‑only dependency rule.

Applying the Architecture in Python – The best practice is to treat Python modules as layers and import statements as dependencies. Example directory structure:

<code>myproject
  __init__.py
  payments/
    __init__.py
    api.py
    vendor.py
  products.py
  shopping_cart.py
</code>

The chosen layer order is:

<code># Dependency flows downward
shopping_cart
payments
products
</code>

To enforce this, imports from payments to shopping_cart are prohibited, while imports from payments to products are allowed.

Further nesting is possible, e.g., within payments :

<code>api
vendor
</code>

Kraken’s Real‑World Implementation – The project is split into three top‑level components:

<code># Dependency flows downward
kraken/
    __init__.py
    client/
        __init__.py
        oede/
        oegb/
        oejp/
        ...
    territories/
        __init__.py
        deu/
        gbr/
        jpn/
        ...
    core/
</code>

Clients sit at the top, each with its own sub‑package; territories follow, providing country‑specific code; core contains shared functionality. Rules ensure client and territory sub‑packages are independent, allowing isolated updates.

Ensuring Compliance with Import Linter – The team uses the open‑source tool Import Linter. A sample contract configuration:

<code>[importlinter:contract:top-level]
name = Top level layers
type = layers
layers =
    kraken.clients
    kraken.territories
    kraken.core
</code>

Additional contracts enforce independence between individual clients and territories. The command lint-import is run on every pull request, preventing merges that violate the defined layer rules.

Technical debt is tracked by counting ignored illegal imports defined in a special contract:

<code>[importlinter:contract:my-layers-contract]
name = My contract
type = layers
layers =
    kraken.clients
    kraken.territories
    kraken.core
ignore_imports =
    kraken.core.customers ->
    kraken.territories.gbr.customers.views
    kraken.territories.jpn.payments -> kraken.utils.urls
    ...
</code>

A burn‑down chart (shown below) visualizes the number of ignored imports over time, demonstrating progress in reducing technical debt.

Drawbacks of Layered Architecture – The approach can lead to excessive high‑level code, increased local complexity when using inversion of control, and the need for ongoing refactoring to eliminate remaining illegal imports (about 15 still persist).

Conclusion – The layered structure enables Kraken to maintain a healthy development workflow despite the massive codebase, limiting dependency chaos and simplifying large‑scale changes. Early adoption of layering reduces future refactoring effort.