Unlocking Python Generators: How They Work Under the Hood

Background

This is a partial translation of the "How Python Generators Work" chapter from 500 Lines or Less . It assumes familiarity with Python functions and the CPython interpreter.

Normal Function Calls

When a regular Python function calls a sub‑routine, the sub‑routine retains control until it returns or raises an exception, after which control returns to the caller.

def foo():
    ...
    bar()
    ...

def bar():
    ...
    pass

CPython Execution Model

The standard CPython interpreter, written in C, uses the PyEval_EvalFrameEx function to execute Python bytecode. It receives a Python stack‑frame object and runs the bytecode in that context. The disassembled bytecode for foo looks like this:

20 LOAD_GLOBAL 0 (bar)
 3 CALL_FUNCTION 0
 6 POP_TOP
 7 LOAD_CONST 0 (None)
10 RETURN_VALUE

During execution, foo loads bar onto the stack, calls it, pops the return value, and finally returns None.

Inspecting Stack Frames

Because stack frames are allocated on the heap, they can survive beyond the lexical call stack. Using the inspect module you can retrieve the current frame and walk back to the caller:

import inspect
frame = None

def foo():
    ...
    bar()
    ...

def bar():
    ...

# inside bar
frame = inspect.currentframe()
caller_frame = frame.f_back
print(frame.f_code.co_name)   # 'bar'
print(caller_frame.f_code.co_name)  # 'foo'

Generators

A generator function is defined with yield. When Python compiles such a function, it sets a generator flag (bit position 5) in the code object’s co_flags:

generator_bit = 1 << 5
bool(gen_fn.__code__.co_flags & generator_bit)   # True

Calling a generator function does not execute its body; instead it returns a generator object:

gen = gen_fn()
type(gen)   # <class 'generator'>

The generator holds a reference to the original code object and its own heap‑allocated stack frame:

gen.gi_code.co_name   # 'gen_fn'

Running a Generator

When send(None) is first called, the generator runs up to the first yield and pauses. The return value of send is the value yielded:

>>> gen.send(None)
1

The instruction pointer advances (e.g., gen.gi_frame.f_lasti == 3) and the bytecode length can be inspected ( len(gen.gi_code.co_code) == 56).

Sending Values and Inspecting Locals

Sending a value replaces the yield expression’s result. After sending 'hello' the generator’s local variable result becomes 'hello':

>>> gen.send('hello')
2
>>> gen.gi_frame.f_locals
{'result': 'hello'}

Completion

A subsequent send('goodbye') drives the generator to its second yield and then raises StopIteration with the return value 'done':

>>> gen.send('goodbye')
Traceback (most recent call last):
  ...
StopIteration: done

The StopIteration exception carries the generator’s final return value.

Visual Aids