Six Common Beginner Mistakes When Using Pandas and How to Avoid Them

We discuss six common beginner mistakes when using Pandas, which are unrelated to API syntax but stem from lack of knowledge and experience.

1. Using pandas.read_csv on very large files is slow; a test on a 1 M‑row dataset took ~22 seconds.

import pandas as pd
%%time
tps_october = pd.read_csv("data/train.csv")
# Wall time: 21.8 s

Solution: use faster I/O libraries such as datatable, Dask, Vaex, or cuDF.

import datatable as dt  # pip install datatable
%%time
tps_dt_october = dt.fread("data/train.csv").to_pandas()
# Wall time: 2 s

2. Not vectorizing operations; loops and apply are slow. Using NumPy vectorized functions can speed up calculations dramatically.

def big_function(col1, col2, col3):
    return np.log(col1 ** 10 / col2 ** 9 + np.sqrt(col3 ** 3))

Applying the function with pandas.apply took ~20 seconds, while using NumPy arrays directly took only 82 ms.

%time tps_october['f1000'] = tps_october.apply(lambda row: big_function(row['f0'], row['f1'], row['f2']), axis=1)
# Wall time: 20.1 s

%time tps_october['f1001'] = big_function(tps_october['f0'].values,
                                         tps_october['f1'].values,
                                         tps_october['f2'].values)
# Wall time: 82 ms

3. Ignoring appropriate dtypes; object dtype consumes most memory. Converting columns to smaller numeric types (int8/16/32, float16/32) can reduce memory usage.

def reduce_memory_usage(df, verbose=True):
    numerics = ["int8","int16","int32","int64","float16","float32","float64"]
    start_mem = df.memory_usage().sum() / 1024**2
    for col in df.columns:
        col_type = df[col].dtypes
        if col_type in numerics:
            c_min = df[col].min()
            c_max = df[col].max()
            # ... (type conversion logic) ...
    end_mem = df.memory_usage().sum() / 1024**2
    if verbose:
        print(f"Mem. usage decreased to {end_mem:.2f} Mb ({100 * (start_mem - end_mem) / start_mem:.1f}% reduction)")
    return df

Applying it to the TPS October dataset reduced memory from 2.2 GB to 509 MB (≈77 % reduction).

>> reduce_memory_usage(tps_october)
Mem. usage decreased to 509.26 Mb (76.9% reduction)

4. Not using DataFrame styling to highlight statistics; pandas.style can add colored bars and gradients without external visualization libraries.

tps_october.sample(20, axis=1).describe().T.style.bar(subset=["mean"], color="#205ff2")\
    .background_gradient(subset=["std"], cmap="Reds")\
    .background_gradient(subset=["50%"], cmap="coolwarm")

5. Saving large DataFrames to CSV is slow; using Feather, Parquet, or pickle is much faster.

%%time
tps_october.to_csv("data/copy.csv")
# Wall time: 2 min 43 s

%%time
tps_october.to_feather("data/copy.feather")
# Wall time: 1.05 s

%%time
tps_october.to_parquet("data/copy.parquet")
# Wall time: 7.84 s

6. Not reading the documentation; many of these tips are covered in pandas’ user guide, especially the “large datasets” section.

In summary, the six mistakes are: using read_csv for large files, not vectorizing, ignoring dtypes, neglecting styling, saving to CSV, and not consulting the documentation. Addressing them helps when working with gigabyte‑scale data and avoids out‑of‑memory errors.