Understanding Python Generators: Functions, Expressions, and Internal Implementation

Introduction

Python generators are implemented via two mechanisms: generator functions and generator expressions. Generator functions use the yield statement to produce a series of values, while generator expressions provide a concise way to create generators.

Generator Functions

Generator functions resemble normal functions but replace return with yield . Calling a generator function does not execute its body immediately; instead, it returns a generator object that implements the iteration protocol ( __iter__ and __next__ ).

Example code:

def count_up_to(n):
    count = 1
    while count <= n:
        yield count
        count += 1
# 创建生成器对象
gen = count_up_to(5)
# 迭代生成器
for number in gen:
    print(number)  # 输出: 1 2 3 4 5

Each call to __next__ resumes execution from the last yield until the next yield or the end of the function, at which point a StopIteration exception is raised.

Generator Expressions

Generator expressions are similar to list comprehensions but use parentheses instead of brackets, producing values lazily on each iteration.

Example code:

squares = (x * x for x in range(5))
# 迭代生成器
for square in squares:
    print(square)  # 输出: 0 1 4 9 16

Internal Implementation – State Machine

The interpreter automatically manages a state machine for generator functions, with states such as initialization (object creation), suspended (when a yield is encountered), resumed (when __next__ or send is called), and completed (raising StopIteration ).

send() Method

Beyond __next__ , generators support send() , allowing a value to be sent into the generator, which becomes the result of the yield expression.

Example code:

def echo_values():
    while True:
        value = yield
        print("Received:", value)

gen = echo_values()
next(gen)  # 初始化生成器
gen.send("Hello")  # 输出: Received: Hello
gen.send("World")  # 输出: Received: World

Low‑Level Implementation in CPython

In CPython, generators are implemented in C as PyGenObject structures that contain a pointer to a state machine represented by bytecode. The bytecode is stored in a PyCodeObject , and the generator object tracks the current instruction pointer, local variables, and evaluation stack.

When a yield instruction executes, the current state is saved and the generator enters a suspended state. Subsequent calls to __next__ or send resume execution from the saved point.

Summary

Generators in Python are special iterators that use yield to defer computation and produce values on demand. They can be created via functions or expressions, managed internally by a state machine, and efficiently implemented in CPython using C structures and bytecode, offering memory‑efficient handling of large data streams.