Unlock Python’s Import System: A Deep Dive into Modules, Meta‑Path, and Hooks

1. Introduction

This article is based on the open‑source project https://github.com/pwwang/python-import-system and expands the explanation so readers can understand Python’s import mechanism in one article.

1.1 What is the import mechanism?

In Python, executing code from another module requires the import mechanism. The import statement is the most common way, but importlib.import_module and the built‑in __import__ function can also be used.

1.2 How does import work?

The import statement performs two steps:

Search for the module to be imported.

Bind the module’s name to a local variable.

The search step is carried out by the built‑in __import__ function, whose return value is then bound to the variable. The following experiment shows what happens when sys.modules contains None for a module.

In [1]: import sys

In [2]: sys.modules['os'] = None

In [3]: import os
---------------------------------------------------------------------------
ModuleNotFoundError                           Traceback (most recent call last)
...
ModuleNotFoundError: import of os halted; None in sys.modules

If the module is already present in sys.modules, the cached module is used; otherwise Python proceeds to a full import process.

2. Overview of the import mechanism

When the import mechanism is triggered, Python first looks in sys.modules. If the module is found, it is returned immediately; otherwise the interpreter walks sys.meta_path to locate a suitable meta‑path finder. The default meta‑path contains three finders: the built‑in finder, the frozen‑module finder, and the path‑based finder.

If a finder returns None for the module, a ModuleNotFoundError is raised. The following interactive session shows the content of sys.meta_path:

In [1]: import sys
In [2]: sys.meta_path
Out[2]:
[_frozen_importlib.BuiltinImporter,
 _frozen_importlib.FrozenImporter,
 _frozen_importlib_external.PathFinder]

Each finder must implement a find_spec method that returns a ModuleSpec object when it can handle the import. If no ModuleSpec is returned, the next finder in sys.meta_path is consulted.

3. Import hooks

Import hooks allow you to modify sys.meta_path or sys.path at runtime. By inserting a custom finder into sys.meta_path, you can change how modules are located. The example below demonstrates a custom finder and loader that create a synthetic module.

import sys
from types import ModuleType
from importlib.machinery import ModuleSpec
from importlib.abc import MetaPathFinder, Loader

class Module(ModuleType):
    def __init__(self, name):
        self.x = 1
        self.name = name

class ExampleLoader(Loader):
    def create_module(self, spec):
        return Module(spec.name)
    def exec_module(self, module):
        module.y = 2

class ExampleFinder(MetaPathFinder):
    def find_spec(self, fullname, path, target=None):
        return ModuleSpec('module', ExampleLoader())

sys.meta_path = [ExampleFinder()]

import module
module
# <module 'module' (<__main__.ExampleLoader object at 0x...>)>
module.x  # 1
module.y  # 2

4. Meta‑path finders

A meta‑path finder's job is to decide whether it can locate a module. The three default finders are the built‑in finder, the frozen‑module finder, and the path‑based finder. The path‑based finder searches a sequence of import locations (file system paths, zip archives, URLs, etc.) for .py, .pyc, .so files, and even zipimport resources.

5. ModuleSpec objects

A ModuleSpec object describes how a module should be loaded. It contains the module's name and a reference to the loader that will create and execute the module. If the loader attribute is None, an ImportError: missing loader is raised.

import sys
sys.__spec__
# ModuleSpec(name='sys', loader=<_frozen_importlib.BuiltinImporter object at 0x...>)

6. Loaders

Loaders implement create_module (to create the module object) and exec_module (to execute the module's code). Most finders act as their own loader; the loader is stored in the loader attribute of the ModuleSpec. If create_module returns None, Python creates a default module object automatically.

7. Summary

The Python import mechanism is flexible and powerful. This article covered the basic steps, the role of sys.modules, meta‑path finders, ModuleSpec, loaders, and how import hooks can be used to customise the process. Details such as sub‑module loading and caching are omitted for brevity.