Cut Pandas DataFrame Memory Usage by 90% with Simple Type Conversions

Introduction

pandas is a Python library for data manipulation and analysis. A Dataquest tutorial demonstrates that simple data type conversions can shrink a baseball game dataset’s memory usage by nearly 90%.

Why Memory Matters

For datasets under 100 MB, performance is usually fine, but larger datasets (hundreds of MB to several GB) can cause slow runtimes and out‑of‑memory failures. While tools like Spark handle massive data, pandas offers richer data‑cleaning capabilities, so optimizing pandas memory is valuable for medium‑sized data.

Loading the Data

import pandas as pd
gl = pd.read_csv('game_logs.csv')
gl.head()

The CSV contains 171,907 rows and 161 columns.

Inspecting Memory Usage

gl.info(memory_usage='deep')

Object columns consume the most memory (≈9.5 MB per column on average), while numeric columns use less.

Understanding Subtypes

Numeric columns are stored as NumPy ndarrays, which use contiguous memory blocks. Subtypes like int8, int16, float16, float32 use fewer bytes per value.

for dtype in ['float','int','object']:
    selected = gl.select_dtypes(include=[dtype])
    mean_mb = selected.memory_usage(deep=True).mean() / 1024**2
    print(f"Average memory usage for {dtype} columns: {mean_mb:.2f} MB")

Object columns average 9.53 MB, far higher than numeric columns.

Downcasting Numeric Columns

def mem_usage(df):
    if isinstance(df, pd.DataFrame):
        usage = df.memory_usage(deep=True).sum()
    else:
        usage = df.memory_usage(deep=True)
    return f"{usage/1024**2:.2f} MB"

gl_int = gl.select_dtypes(include=['int'])
converted_int = gl_int.apply(pd.to_numeric, downcast='unsigned')
print(mem_usage(gl_int))
print(mem_usage(converted_int))

Integer columns shrink from 7.87 MB to 1.48 MB (≈80% reduction).

gl_float = gl.select_dtypes(include=['float'])
converted_float = gl_float.apply(pd.to_numeric, downcast='float')
print(mem_usage(gl_float))
print(mem_usage(converted_float))

Float columns shrink from 100.99 MB to 50.49 MB (≈50% reduction).

Using Categoricals for Object Columns

Object columns store Python strings, leading to high memory usage. Converting low‑cardinality string columns to category dtype stores values as integer codes plus a mapping dictionary.

dow = gl['day_of_week']
dow_cat = dow.astype('category')
print(dow_cat.head())
print(dow_cat.cat.codes.head())

Memory for this column drops from 9.84 MB to 0.16 MB (≈98% reduction).

gl_obj = gl.select_dtypes(include=['object']).copy()
converted_obj = pd.DataFrame()
for col in gl_obj.columns:
    if gl_obj[col].nunique() / len(gl_obj[col]) < 0.5:
        converted_obj[col] = gl_obj[col].astype('category')
    else:
        converted_obj[col] = gl_obj[col]
print(mem_usage(gl_obj))
print(mem_usage(converted_obj))

Object columns shrink from 752 MB to 52 MB (≈93% reduction).

Optimizing While Reading

After determining optimal dtypes, they can be passed to pd.read_csv via the dtype argument, and dates parsed with parse_dates.

column_types = {...}  # dictionary of column name → dtype (e.g., 'category', 'float32')
read_opt = pd.read_csv('game_logs.csv', dtype=column_types, parse_dates=['date'], infer_datetime_format=True)
print(mem_usage(read_opt))

The resulting DataFrame uses only 104 MB, an 88% reduction from the original 861 MB.

Exploratory Analysis

With the optimized DataFrame, simple analyses are performed, such as visualizing game counts by day of week over years and game length trends.

optimized_gl['year'] = optimized_gl.date.dt.year
games_per_day = optimized_gl.pivot_table(index='year', columns='day_of_week', values='date', aggfunc='size')
games_per_day.plot(kind='area', stacked=True)

Plots show the rise of Sunday games after the 1920s and increasing game lengths since the 1940s.

Conclusion

By downcasting numeric columns, converting low‑cardinality object columns to categoricals, and specifying optimal dtypes during CSV import, pandas memory usage can be reduced from 861 MB to about 104 MB—a reduction of roughly 88%.