Python Memory Optimization Techniques: __slots__, Generators, mmap, Data Types, and String Interning

When projects grow, managing memory becomes essential; Python may appear less memory‑efficient than C/C++, but many built‑in mechanisms can dramatically reduce its footprint.

Inspecting memory usage can be done with sys.getsizeof(), objgraph.show_refs(), or psutil.Process().memory_info().rss. Example:

>> import numpy as np
>>> import sys
>>> import objgraph
>>> import psutil
>>> import pandas as pd
>>> ob = np.ones((1024, 1024, 1024, 3), dtype=np.uint8)
>>> sys.getsizeof(ob) / (1024 * 1024)
3072.0001373291016
>>> psutil.Process().memory_info().rss / (1024 * 1024)
3234.19140625
>>> objgraph.show_refs([ob], filename='sample-graph.png')
>>> from sklearn.datasets import load_boston
>>> data = load_boston()
>>> data = pd.DataFrame(data['data'])
>>> print(data.info(verbose=False, memory_usage='deep'))
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 506 entries, 0 to 505
Columns: 13 entries, 0 to 12
memory usage: 51.5 KB
>>> data[0].memory_usage(deep=True)
4176

Using __slots__ removes the per‑instance __dict__, saving memory. Without slots:

class Author:
    def __init__(self, name, age):
        self.name = name
        self.age = age

me = Author('Yang Zhou', 30)
me.job = 'Software Engineer'
print(me.job)  # Software Engineer

With __slots__:

class Author:
    __slots__ = ('name', 'age')
    def __init__(self, name, age):
        self.name = name
        self.age = age

me = Author('Yang Zhou', 30)
me.job = 'Software Engineer'  # AttributeError

Memory comparison shows a reduction from 152 bytes to 48 bytes per instance.

Generators provide lazy evaluation, yielding items on demand and thus using far less memory than full lists. Example:

def number_generator():
    for i in range(100):
        yield i

numbers = number_generator()
print(next(numbers))  # 0
print(next(numbers))  # 1

Size comparison:

numbers = []
for i in range(100):
    numbers.append(i)

def number_generator():
    for i in range(100):
        yield i
numbers_generator = number_generator()
print(sys.getsizeof(numbers_generator))  # 112
print(sys.getsizeof(numbers))           # 920

Memory‑mapped files (mmap) allow large files to be accessed without loading them entirely into RAM:

import mmap
with open('test.txt', "r+b") as f:
    with mmap.mmap(f.fileno(), 0) as mm:
        print(mm.read())
        snippet = mm[0:10]
        print(snippet.decode('utf-8'))

Choosing appropriate data types also saves memory: tuples are smaller than lists, and arrays are smaller than lists of homogeneous numbers.

my_tuple = (1, 2, 3, 4, 5)
my_list = [1, 2, 3, 4, 5]
print(sys.getsizeof(my_tuple))  # 80
print(sys.getsizeof(my_list))   # 120

import array
my_list = [i for i in range(1000)]
my_array = array.array('i', [i for i in range(1000)])
print(sys.getsizeof(my_list))   # 8856
print(sys.getsizeof(my_array))  # 4064

String interning reuses identical small strings, reducing memory usage. Python automatically interns strings up to 4096 characters; longer strings can be interned manually with sys.intern():

>> a = 'Y'*4096
>>> b = 'Y'*4096
>>> a is b
True
>>> c = 'Y'*4097
>>> d = 'Y'*4097
>>> c is d
False
>>> import sys
>>> c = sys.intern('Y'*4097)
>>> d = sys.intern('Y'*4097)
>>> c is d
True

By applying these techniques—monitoring usage, using __slots__, generators, mmap, efficient containers, and string interning—developers can significantly reduce Python memory consumption.