5 Python Memory‑Optimization Patterns That Cut Usage by 70%

When processing gigabyte‑scale data, the author’s Python program repeatedly crashed due to excessive memory use. After weeks of profiling with memory_profiler and tracemalloc, a series of refactorings reduced peak memory by roughly 70% and made the code faster.

1. Stream large files instead of loading them all at once

Instead of reading an entire file into a list, read it in fixed‑size chunks and process each chunk immediately.

# Inefficient: load whole file
with open('huge.csv') as f:
    data = f.readlines()  # consumes all RAM

# Efficient: stream in chunks
def read_in_chunks(file_path, chunk_size=1024*1024):
    """Yield file chunks to avoid full load"""
    with open(file_path, 'rb') as f:
        while chunk := f.read(chunk_size):
            yield chunk

for chunk in read_in_chunks('huge.csv'):
    process(chunk)

Reading a 1 GB file this way drops peak memory from gigabytes to only a few megabytes, allowing the author to handle a 7 GB log on an 8 GB laptop.

2. Replace large lists with generator expressions

Building a list of a million items consumes the full memory for all elements, while a generator computes items lazily.

# List comprehension (high memory)
items = [expensive_function(x) for x in range(1_000_000)]
for item in items:
    do_something(item)

# Generator expression (low memory)
items = (expensive_function(x) for x in range(1_000_000))
for item in items:
    do_something(item)

Switching to a generator lowered the peak RAM from 3.4 GB to 280 MB.

3. Use __slots__ to create lean objects

Normal Python objects store attributes in a per‑instance __dict__, which adds overhead. Declaring __slots__ fixes the attribute layout and removes the dict.

# Regular class
class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

# Slot‑based class
class Point:
    __slots__ = ('x', 'y')
    def __init__(self, x, y):
        self.x = x
        self.y = y

For ten million tiny objects, __slots__ saved about 500 MB. Benchmarks showed a 46.7% memory reduction, 37.5% faster object creation, and a 4.8% speed‑up in attribute access. The trade‑off is that new attributes cannot be added at runtime.

4. Avoid creating temporary objects inside loops

Appending results to a list creates many short‑lived objects. Using a generator with yield produces each result on demand.

# Inefficient: accumulate results
results = []
for row in big_dataset:
    results.append(process(row))

# Efficient: generator pipeline
def process_dataset(dataset):
    """Yield processed rows one by one"""
    for row in dataset:
        yield process(row)

for result in process_dataset(big_dataset):
    handle(result)

This keeps memory stable regardless of input size.

5. Reuse a buffer instead of repeatedly reallocating

Repeated string concatenation creates new objects each time. A BytesIO buffer grows in place.

# Naïve concatenation (high allocation)
data = b''
for chunk in stream:
    data += chunk  # creates a new bytes object each iteration

# Better: reusable buffer
from io import BytesIO
buffer = BytesIO()
for chunk in stream:
    buffer.write(chunk)
data = buffer.getvalue()

In a real workload this cut peak allocation by 62% and eliminated periodic GC‑induced freezes.

Advanced considerations

Understanding Python’s internal object cache and measuring hotspots with memory_profiler and tracemalloc is essential; typically 90% of memory usage originates from 10% of the code. Profiling first, then applying the above patterns, yields the best results.

Conclusion

Stream large files instead of loading them entirely.

Prefer generator expressions for lazy evaluation.

Apply __slots__ to classes with many instances.

Avoid accumulating temporary objects; use generators.

Reuse buffers (e.g., BytesIO) rather than repeatedly reallocating.

Before optimizing, profile with memory_profiler and tracemalloc to locate the true bottlenecks.