Master Python Coroutines: From Basics to Real‑World File Search

1. Coroutine Basics

Coroutines, also called micro‑threads or fibers, are lightweight user‑level threads. Unlike system‑level threads scheduled by the OS, coroutines are scheduled by the programmer within a single thread, allowing a function to pause its execution, switch to another function, and later resume from the exact point it left off.

Technically, a coroutine has its own register context and stack; when switched, these are saved and restored, preserving the state of the previous call.

1.1 Advantages

No OS thread‑context switching overhead, leading to higher performance (though the programmer must manage scheduling).

No need for atomic operations, locks, or synchronization.

Simplifies control‑flow and programming model.

Supports massive concurrency on a single CPU, making it suitable for high‑concurrency scenarios.

1.2 Disadvantages

Cannot utilize multiple CPU cores directly; coroutines run in a single thread.

Blocking operations (e.g., I/O) block the entire program.

2. Implementing Coroutines in Python

2.1 Using yield to create a coroutine

def eater(name):
    print("%s eat food" % name)
    while True:
        food = yield
    print("done")

g = eater("gangdan")
print(g)

Output:

<generator object eater at 0x0000000002140FC0>

2.2 Sending values into a coroutine

def creater(name):
    print("%s start to eat food" % name)
    food_list = []
    while True:
        food = yield food_list
        print("%s get %s ,to start eat" % (name, food))
        food_list.append(food)

builder = creater('tom')
next(builder)
print(builder.send('包子'))
print(builder.send('骨头'))
print(builder.send('菜汤'))

Output demonstrates that next() must be called first to advance the generator to the yield point before send() can provide a value.

2.3 Decorating coroutines to auto‑initialize

def init(func):
    def wrapper(*args, **kwargs):
        builder = func(*args, **kwargs)
        next(builder)  # automatically advance to first yield
        return builder
    return wrapper

@init
def creater(name):
    print("%s start to eat food" % name)
    food_list = []
    while True:
        food = yield food_list
        print("%s get %s ,to start eat" % (name, food))
        food_list.append(food)

Now the coroutine is ready for send() without an explicit next() call.

3. Practical Application: Grep‑like File Search Using Generators

The example builds a pipeline of generators to locate files, open them, read lines, filter by a pattern, and print matching file names.

3.1 Generator that yields absolute file paths

@init
def search():
    while True:
        dir_name = yield
        for dir_path, _, files in os.walk(dir_name):
            for file in files:
                abspath = f"{dir_path}\{file}"
                target.send(abspath)

3.2 Generator that opens a file and yields its handle

@init
def opener(target):
    while True:
        abspath = yield
        with open(abspath, 'rb') as f:
            target.send((abspath, f))

3.3 Generator that reads lines from a file

@init
def cat(target):
    while True:
        abspath, f = yield
        for line in f:
            res = target.send((abspath, line))
            if res:
                break

3.4 Generator that filters lines containing a pattern

@init
def grep(pattern, target):
    tag = False
    while True:
        abspath, line = yield tag
        tag = False
        if pattern in line:
            target.send(abspath)
            tag = True

3.5 Generator that prints matching file paths

@init
def printer():
    while True:
        abspath = yield
        print(abspath)

Pipeline composition:

g = search(opener(cat(grep(b'error', printer()))))
g.send(r'E:\Python\script')

The pipeline prints the absolute paths of files containing the word "error".

— End of tutorial —