Master Python Generators (And the Odd Feud Behind It)

Generator basics

When a function contains the yield keyword it becomes a generator function; calling it returns a generator object. The generator yields values lazily, which avoids loading an entire large file into memory.

def read_file(file):
    return open(file, encoding="utf-8").readlines()

print(read_file("big.txt"))
# ['line1
', 'line2
', ...]

Using yield turns the same logic into a lazy iterator:

from typing import Iterator, Generator

def read_file(file) -> Iterator[str]:
    with open(file, encoding="utf-8") as f:
        for line in f:
            yield line

for line in read_file("big.txt"):
    print(line, end="")

Driving a generator: __next__

The generator is a special iterator. Calling __next__() (or next()) advances execution to the next yield and returns the yielded value. When no further yield exists a StopIteration is raised, optionally carrying a return value.

def gen():
    yield 123
    yield 456
    yield 789
    return "result"

g = gen()
print(g.__next__())  # 123
print(g.__next__())  # 456
print(g.__next__())  # 789
try:
    g.__next__()
except StopIteration as e:
    print(e.value)   # result

Sending values: send

send(value)

both resumes the generator and injects value as the result of the paused yield expression.

def gen():
    res1 = yield "yield 1"
    print(f"***** {res1} *****")
    res2 = yield "yield 2"
    return res2

g = gen()
print(g.__next__())          # yield 1
print(g.send("hey"))        # ***** hey *****
 yield 2
try:
    g.send("oops")
except StopIteration as e:
    print(e.value)            # oops

Throwing exceptions: throw

throw(exc)

injects an exception at the current yield. If the generator catches it, execution continues; otherwise the exception propagates to the caller.

def gen():
    try:
        yield 123
    except ValueError as e:
        print(f"异常：{e}")
    yield 456
    return "result"

g = gen()
g.__next__()                     # 123
g.throw(ValueError("bad"))        # 异常：bad
print(g.__next__())                # 456

Generator pre‑activation and frame state

When a generator is created its internal frame attribute f_lasti is -1, meaning no bytecode has executed yet. In Python 3.12 this field was replaced by gi_frame_state with five possible values:

FRAME_CREATED (-2)

FRAME_SUSPENDED (-1)

FRAME_EXECUTING (0)

FRAME_COMPLETED (1)

FRAME_CLEARED (4)

def gen():
    yield 123
    yield 456
    return "result"

g = gen()
print(g.gi_frame.f_lasti)  # -1 (pre‑activation)

Closing a generator and GeneratorExit

Calling close() forces the generator to raise GeneratorExit inside. If the generator catches this exception it can perform cleanup, but after handling it must not yield again; otherwise a RuntimeError is raised.

def gen():
    try:
        yield 123
    except GeneratorExit:
        print("generator deleted")
    return "result"

g = gen()
g.__next__()   # 123
g.close()       # prints "generator deleted"

If close() is called without catching GeneratorExit, the generator simply terminates and subsequent __next__ raises StopIteration with value=None.

Yield‑from delegation

yield from iterable

requires the right‑hand side to be an iterable. It yields each element of the iterable, forwards sent values and exceptions, and captures the sub‑generator’s return value.

def gen():
    yield from [1, 2, 3]
    return "result"

g = gen()
print(g.__next__())  # 1
print(g.__next__())  # 2
print(g.__next__())  # 3
try:
    g.__next__()
except StopIteration as e:
    print(e.value)   # result

Delegation creates a two‑way channel: values sent to the delegating generator are passed to the sub‑generator, and exceptions raised in the sub‑generator can be caught by the delegating generator.

def sub():
    try:
        yield 123
    except ValueError as e:
        print(f"sub caught: {e}")
    return "sub result"

def delegator():
    try:
        res = yield from sub()
    except ValueError as e:
        yield f"delegator caught: {e}"
    yield f"sub returned: {res}"

g = delegator()
print(g.__next__())                     # 123
print(g.throw(ValueError("oops")))    # sub caught: oops
print(g.__next__())                     # sub returned: sub result

Generator expressions and variable capture pitfalls

Generator expressions are lazy comprehensions written as (x for x in iterable). The iterable after in is bound at creation time, while other free variables are looked up when the generator runs.

i = 1
g = (x + i for x in [1, 2, 3])
i = 10
print(tuple(g))  # (11, 12, 13)

Re‑binding the iterable after the generator is created does not affect the generator’s output because the iterable reference is captured at creation:

lst = [1, 2, 3]
g = (x for x in lst)
lst = [4, 5, 6]
print(tuple(g))  # (1, 2, 3)

Generators as coroutines

Before native async/await, many asynchronous frameworks (e.g., Tornado) implemented coroutines with generators. Native coroutines are also built on top of generator mechanics: an async def function returns a coroutine object whose __await__ method yields control points.

async def native_coroutine():
    return "result"

try:
    native_coroutine().__await__().__next__()
except StopIteration as e:
    print(e.value)   # result

Mixing a native coroutine with a generator‑based coroutine using asyncio.gather yields identical results and execution time.

import asyncio, time, types

async def some_task():
    await asyncio.sleep(3)
    return "task result"

async def native_coroutine():
    r = await some_task()
    return f"{r} from native coroutine"

@types.coroutine
def generator_coroutine():
    r = yield from some_task()
    return f"{r} from generator coroutine"

async def main():
    start = time.time()
    result = await asyncio.gather(native_coroutine(), generator_coroutine())
    end = time.time()
    print(result)
    print(f"耗时：{end - start}")

asyncio.run(main())
# ['task result from native coroutine', 'task result from generator coroutine']
# 耗时：3.00...

Summary of core concepts

Generator functions use yield to produce values lazily. __next__, send and throw drive execution and interact with the paused yield.

Pre‑activation state can be inspected via gi_frame.f_lasti (or gi_frame_state in 3.12). close() raises GeneratorExit; catching it allows resource cleanup, but yielding after catching is illegal. yield from delegates to a sub‑generator, forwards values/exceptions, and captures the sub‑generator’s return value.

Generator expressions are lazy but capture free variables at execution time, which can lead to surprising results.

Before async/await, generators were the foundation of coroutine implementations; native coroutines still rely on the same underlying protocol.