How Python Generators Work: From Yield to Bytecode Execution

Generator Basics

The yield keyword works like return but returns a generator object instead of a final value.

A function containing yield becomes a generator; each call to the generator returns one value and preserves the function's execution state for the next call.

Generators support the next method to retrieve the next value.

Basic Operations

# Create a generator using yield <code>def func(): for i in xrange(10): yield i </code> # Create a generator using a list comprehension <code>[i for i in xrange(10)] </code> # Using the generator <code>f = func() print(f) # <em>generator object</em> print(next(f)) # 0 print(next(f)) # 1 # ... print(next(f)) # 9 </code> The generator raises StopIteration after the last value.

Sending Values

Generators also support the send method to pass a value back into the generator.

<code>def func(): n = 0 while True: n = yield n </code> <code>f = func() next(f) # start generator, returns None f.send(1) # n becomes 1, yields 1 f.send(2) # n becomes 2, yields 2 </code>

Practical Example: Infinite Sequence

Generating an infinite prime sequence without storing all numbers in memory:

def get_primes(start):
    for element in magical_infinite_range(start):
        if is_prime(element):
            return element

Using a generator avoids the memory limit of a large list.

Generator Source Analysis

The CPython implementation resides in Objects/genobject.c. Understanding the call stack is essential.

Call Stack

Python's virtual machine uses a stack of PyFrameObject structures (defined in Include/frameobject.h) to keep track of execution context, code objects, locals, globals, and exception state.

Generator Creation

PyObject *PyGen_New(PyFrameObject *f) {
    PyGenObject *gen = PyObject_GC_New(PyGenObject, &PyGen_Type);
    if (gen == NULL) {
        Py_DECREF(f);
        return NULL;
    }
    gen->gi_frame = f;
    Py_INCREF(f->f_code);
    gen->gi_code = f->f_code;
    gen->gi_running = 0;
    gen->gi_weakreflist = NULL;
    _PyObject_GC_TRACK(gen);
    return (PyObject *)gen;
}

send and next

Both next and send call the internal function gen_send_ex. The difference is whether a value is passed.

static PyObject *gen_iternext(PyGenObject *gen) {
    return gen_send_ex(gen, NULL, 0);
}

static PyObject *gen_send(PyGenObject *gen, PyObject *arg) {
    return gen_send_ex(gen, arg, 0);
}

gen_send_ex

checks if the generator is already running, handles the first‑call restrictions (no non‑None argument), pushes the argument onto the frame's value stack, updates the thread state, executes the frame with PyEval_EvalFrameEx, and restores the running flag.

Bytecode Execution

The core evaluation loop in PyEval_EvalFrameEx fetches opcodes and dispatches via a switch. When the opcode is YIELD_VALUE, the interpreter pops the top of the stack, records the reason for yielding, updates the frame's stack pointer, and returns the yielded value.

case YIELD_VALUE:
    retval = POP();
    f->f_stacktop = stack_pointer;
    why = WHY_YIELD;
    goto fast_yield;

Disassembly Example

Using the dis module to inspect bytecode of a generator function shows the sequence of opcodes, including YIELD_VALUE and RETURN_VALUE, and demonstrates how f_lasti and f_back change during execution.

import sys
from dis import dis

def func():
    f = sys._getframe(0)
    print(f.f_lasti)
    print(f.f_back)
    yield 1
    print(f.f_lasti)
    print(f.f_back)
    yield 2

a = func()
print(a)
print(a.next())
print(a.next())

The disassembly output confirms the YIELD_VALUE opcode at the appropriate offsets.

Source: cococo点点 (http://www.cnblogs.com/coder2012/p/4990834.html)