Master Python Iterators and Generators: From List Comprehensions to Efficient Memory Use

Python Iterators and Generators

List Comprehensions

List comprehensions (also called list comprehensions) produce a list based on a defined rule.

list2 = [x for x in range(10)]
print(list2)

A more complex example:

list1 = [x * y for x in range(1, 5) if x > 2 for y in range(1, 4) if y < 3]
# Equivalent expanded code:
list4 = []
for x in range(1, 5):
    if x > 2:
        for y in range(1, 4):
            if y < 3:
                list4.append(x * y)

Generators

Creating a list with a million elements using a list comprehension consumes a lot of memory; a generator stores only the rule and generates elements on demand, reducing memory usage, though it can only be iterated sequentially.

Define a generator by replacing brackets with parentheses: generator = (x for x in range(10)) A generator is a special iterator:

print("Is generator an iterator:", isinstance(generator, Iterator))  # True
print("Is generator iterable:", isinstance(generator, Iterable))  # True

Elements are retrieved with next() (or generator.__next__()) and raise StopIteration when exhausted.

print(generator.__next__())  # or next(generator)

Example: Fibonacci sequence generator:

def fib(length):
    """Generate Fibonacci numbers up to the given length."""
    n, a, b = 0, 0, 1
    while n < length:
        yield b
        a, b = b, a + b
        n += 1
    return 'done'

for i in fib(10):
    print(i)

Iterators

An iterator is an object that returns the next element via next(). Not all iterable collections (list, dict, str) are iterators, but they can be converted.

list1 = [1, 2, 3, 4, 5, 6]
print("Is list1 an iterator:", isinstance(list1, Iterator))  # False
print("Is list1 iterable:", isinstance(list1, Iterable))  # True
print("Length of list1:", len(list1))

if hasattr(list1, "__next__"):
    print("list1 is an iterator")

Convert a list to an iterator:

interObj = iter(list1)
print("Is interObj an iterator:", isinstance(interObj, Iterator))  # True
print("Is interObj iterable:", isinstance(interObj, Iterable))  # False
if hasattr(interObj, "__next__"):
    print("interObj is an iterator")

Iterable objects are not necessarily iterators, but they are always iterable.

Iterators are always iterable; generators are a special kind of iterator.

The main difference is that iterables support len() while iterators do not.

Iterators provide a next() method; not all iterables do.