How Layered Architecture Keeps a 28K‑File Python Monolith Manageable

TLDR (AI Claude’s Summary)

This article, written by a Python developer, describes the layered architecture used to manage a Python project containing nearly 30,000 files maintained by over 400 developers worldwide.

Introduction

David, a Python developer at Kraken Technologies, maintains a Python application with 27,637 modules (about 28K independent files, not counting tests). The code runs for 17 energy and utility companies serving millions of customers. Despite its size, the project remains maintainable thanks to cultural rules and a well‑designed code organization.

Layered Architecture

As codebases grow, logic from different parts becomes tangled, making it hard to understand individual modules. Kraken adopted a layered architecture: the codebase is divided into components (layers) with dependencies allowed only from higher layers to lower layers.

In a layered system, lower‑level components cannot depend on higher‑level ones. The diagram below illustrates a downward‑only dependency flow.

Components can be defined flexibly—either as independent services or as groups of source files. Any direct cross‑reference between components is considered a dependency; indirect references are usually ignored.

Applying Layered Architecture in a Python Project

The best practice is to treat Python modules as layers and import statements as dependencies. For example, given a repository structure:

myproject
  __init__.py
  payments/
    __init__.py
    api.py
    vendor.py
  products.py
  shopping_cart.py

We might define layers in the order:

# Dependencies flow downward (higher can depend on lower)
shopping_cart
payments
products

This means modules in payments must not import from shopping_cart, but may import from products. Layers can be nested, e.g., further subdividing payments into api and vendor.

The exact number and ordering of layers depend on practical experience, but a well‑applied layered architecture reduces structural complexity and improves understandability.

Kraken’s Real‑World Layered Structure

Kraken serves multiple client instances, each with its own custom code. The top‑level client component contains a sub‑package per client (e.g., oede for Octopus Energy Germany). Below that, territories holds region‑specific code, and the bottom core component contains shared logic.

# Dependency flow downward
kraken/
  __init__.py
  client/
    __init__.py
    oede/
    oegb/
    oejp/
    ...
  territories/
    __init__.py
    deu/
    gbr/
    jpn/
    ...
  core/

Rules enforce that sub‑packages under client and territories are independent and cannot be imported by other clients or territories. This isolation allows developers to modify a single client or region without affecting others, enabling fast, independent cross‑team development.

Ensuring Layer Compliance with Import Linter

To prevent accidental violations, Kraken uses the open‑source tool Import Linter. A configuration file defines the allowed layers:

[importlinter:contract:top-level]
name = Top level layers
type = layers
layers =
    kraken.clients
    kraken.territories
    kraken.core

Additional contracts enforce independence between specific clients and territories:

# Client independence
[importlinter:contract:client-independence]
name = Client independence
type = independence
layers =
    kraken.clients.oede
    kraken.clients.oegb
    kraken.clients.oejp
    ...

# Territory independence
[importlinter:contract:territory-independence]
name = Territory independence
type = independence
layers =
    kraken.territories.deu
    kraken.territories.gbr
    kraken.territories.jpn
    ...

Running lint-import during CI flags any illegal imports, preventing non‑compliant code from being merged. Kraken maintains over 40 such configuration files to govern its architecture.

Managing Technical Debt

Import Linter also supports ignoring specific imports while they are being refactored. Kraken tracks the number of ignored imports as a metric of technical debt, visualizing progress over time.

[importlinter:contract:my-layers-contract]
name = My contract
type = layers
layers =
    kraken.clients
    kraken.territories
    kraken.core
ignore_imports =
    kraken.core.customers ->
    kraken.territories.gbr.customers.views
    kraken.territories.jpn.payments -> kraken.utils.urls
    (and so on...)

The chart shows the number of ignored problematic imports since May 1 2022, illustrating the team’s ongoing effort to reduce debt.

Drawbacks of Layered Architecture

Complex Reality

Real‑world projects have tangled dependencies, and developers sometimes need to break layer boundaries. Techniques like Inversion of Control can help, though they add local complexity.

Excess High‑Level Code

Higher layers tend to accumulate more code because they are easier to change. Core code changes are costly and risky, encouraging developers to add functionality in higher layers, which can lead to an overabundance of top‑level code.

Incomplete Debt Resolution

Even after years of effort, some ignored imports remain unresolved, highlighting the difficulty of fully refactoring a massive codebase.

Conclusion

Kraken’s layered structure enables healthy development and maintenance of a huge Python monolith, keeping the difficulty of operations low despite the scale. Enforcing dependency rules prevents the codebase from becoming a tangled mess, and early adoption of layering reduces future refactoring effort.

If you are building a large Python project—or even a smaller one—consider trying a layered architecture early; the sooner you adopt it, the fewer headaches you’ll face later.