How Layered Architecture Keeps a 30K‑File Python Codebase Manageable

TLDR (AI Claude summary)

This article is a Python developer's summary of a massive Python project's code structure.

The project contains nearly 30,000 Python files maintained by over 400 developers worldwide. To handle growing complexity, it adopts a layered architecture that partitions the codebase into multiple layers and restricts dependencies to flow only from higher to lower layers.

The article details the layered structure and how the Import Linter tool enforces the layering rules by tracking ignored illegal import statements.

While layering effectively reduces complexity and facilitates independent development, it has drawbacks such as potential code bloat in higher layers and the time required for full implementation. Introducing layering early can reduce later refactoring effort, making it an effective strategy for managing large Python projects.

Introduction

Hello, I’m David, a Python developer at Kraken Technologies. I help maintain a Python application with 27,637 modules—nearly 28 K independent Python files (excluding tests). Over 400 developers collaborate on this massive codebase, deploying updates across 17 energy and utility companies serving millions of customers.

Despite its size, the project remains manageable thanks to a well‑designed code organization.

Layered Architecture

As a codebase grows, modules become intertwined, making it harder to understand and modify individual parts. To address this, we adopted a layered architecture: the codebase is divided into components (layers) and dependencies are only allowed from higher layers to lower ones.

In a layering model, lower‑level components cannot depend on higher‑level ones. The diagram below illustrates a dependency flow where component C can depend on B and A but not on D.

Components can be defined flexibly—either as deployable services or as groups of source files. Any direct cross‑component import is considered a dependency; indirect configuration‑based interactions are usually ignored.

Applying Layering in a Python Project

The best practice is to use Python modules as the layering basis and treat import statements as dependency indicators.

Example directory structure:

myproject
  __init__.py
  payments/
    __init__.py
    api.py
    vendor.py
  products.py
  shopping_cart.py

We define the layer order as:

# Dependency flows downwards (higher layers can import lower layers)
shopping_cart
payments
products

To enforce this, imports from shopping_cart into payments are prohibited, while imports from products into payments are allowed.

Layers can be nested; for example, the payments module can be further split into api and vendor sub‑layers.

Kraken’s Real‑World Layering Implementation

Kraken serves 17 different clients, each with its own instance. The top‑level client component contains a sub‑package per client (e.g., oede for Octopus Energy Germany). Below that, the territories component groups country‑specific code, and the bottom core component holds shared functionality.

# Dependency flows downwards
kraken/
  __init__.py
  client/
    __init__.py
    oede/
    oegb/
    oejp/
    ...
  territories/
    __init__.py
    deu/
    gbr/
    jpn/
    ...
  core/

Clients and territories are required to be independent—no component may import another client or another territory.

This structure allows isolated updates: changes in a specific client or territory do not affect others, enabling rapid, independent development across teams.

Ensuring Layering with Import Linter

To prevent accidental violations, we use the open‑source tool Import Linter. A contract is defined in an INI file, for example:

[importlinter:contract:top-level]
name = Top level layers
type = layers
layers =
    kraken.clients
    kraken.territories
    kraken.core

Additional contracts enforce independence between individual clients and between territories:

# File 1
[importlinter:contract:client-independence]
name = Client independence
type = independence
layers =
    kraken.clients.oede
    kraken.clients.oegb
    kraken.clients.oejp
    ...

# File 2
[importlinter:contract:territory-independence]
name = Territory independence
type = independence
layers =
    kraken.territories.deu
    kraken.territories.gbr
    kraken.territories.jpn
    ...

Running lint-import reports any import that breaches the contracts; the check runs on every pull request, preventing non‑compliant code from being merged.

We have over 40 configuration files covering various sub‑layers.

Tracking Technical Debt

Import Linter allows us to ignore certain imports temporarily. The number of ignored imports serves as a metric for technical debt reduction over time.

[importlinter:contract:my-layers-contract]
name = My contract
type = layers
layers =
    kraken.clients
    kraken.territories
    kraken.core
ignore_imports =
    kraken.core.customers ->
    kraken.territories.gbr.customers.views
    kraken.territories.jpn.payments -> kraken.utils.urls
    (and so on...)

The chart shows the decline of ignored illegal imports over more than a year, illustrating progress in reducing technical debt.

Drawbacks of Layered Architecture

Complex Reality

Real‑world projects have tangled dependencies, and breaking layer rules can be tempting. Inversion of Control (IoC) can resolve such cases but adds local complexity.

Excessive High‑Level Code

Higher layers tend to accumulate more code because they are easier to change, while modifications to the low‑level core are risky and costly.

Incomplete Migration

Even after years, some ignored imports remain unresolved (at least 15 stubborn cases), highlighting the effort required to fully refactor a large codebase.

Conclusion

Kraken’s layered structure enables healthy development and maintenance of a massive codebase with relatively low operational difficulty. Without strict dependency limits, the repository would become an unmanageable tangled mess. Early adoption of layering reduces future refactoring effort, making large‑scale Python projects feasible.