Pandas vs Polars vs DuckDB: Benchmarking Performance on a 1.2M‑row CSV

Test Setup: Three Tools Under the Same Workload

The author used a single 2.3 GB CSV file containing about 1.2 million rows and ran identical operations on Pandas, Polars, and DuckDB on the same DigitalOcean Droplet (4 vCPU, 8 GB RAM). The workload consisted of reading the file, filtering rows where revenue > 1000, grouping by region and quarter, aggregating revenue sum and transaction count, joining with a 50 k‑row lookup table, and sorting the result by total revenue descending.

Pandas

Pandas loads the entire CSV into memory, converting strings to Python objects (≈49 bytes each) and up‑casting numeric columns, which quickly inflates memory usage. Reading alone took 47 seconds; during groupby memory rose to 6.8 GB, and the merge step peaked at 7.2 GB, causing the OS to swap and the OOM killer to terminate the process. After manually specifying dtypes (e.g., using category for strings and float32 for floats) memory dropped to ~4.5 GB and total runtime fell to 89 seconds, but the solution proved fragile—any unexpected string in a numeric column caused a crash. Scaling to 1 billion rows crashes even on a 64 GB machine.

Polars

Polars is written in Rust, uses the Apache Arrow columnar format, enables multithreading by default, and provides a lazy evaluation engine that builds a query plan before execution. On the same machine it read the CSV in 3.2 seconds, and the full pipeline finished in 8.7 seconds with a peak memory usage of 1.8 GB (about 60 % less than Pandas). The lazy API postpones execution until .collect() is called, allowing filter push‑down and column pruning. Using .collect(engine="streaming") enables chunked streaming. An external benchmark from H2O.ai reported a groupby time of 0.45 seconds for Polars versus 12.5 seconds for Pandas (28× faster). The author notes a learning curve because Polars’ expression API differs from Pandas (e.g., df['col'].mean() becomes pl.col('col').mean()).

import polars as pl
result = (
    pl.scan_csv("data.csv")
    .filter(pl.col("revenue") > 1000)
    .group_by(["region", "quarter"])
    .agg([
        pl.col("revenue").sum(),
        pl.col("transactions").count()
    ])
    .join(lookup, on="region_id")
    .sort("revenue_sum", descending=True)
    .collect()
)

DuckDB

DuckDB is an in‑process analytical database that executes SQL directly in the Python process without a server. The same query runs in 12 milliseconds (repeated runs gave 11 ms and 13 ms). DuckDB compiles SQL to an optimized, vectorized execution plan, reads only required columns, and spills to disk when data exceeds memory. In the TPC‑H benchmark DuckDB completed the full suite on a single machine in 1 minute 16 seconds, whereas Apache Spark needed about 8 minutes on a 32‑node cluster. DuckDB can also query Pandas or Polars DataFrames directly:

import duckdb
result = duckdb.sql("""
    SELECT d.region, d.quarter,
           SUM(d.revenue) AS revenue_sum,
           COUNT(*) AS transaction_count
    FROM read_csv_auto('data.csv') d
    JOIN lookup l ON d.region_id = l.region_id
    WHERE d.revenue > 1000
    GROUP BY d.region, d.quarter
    ORDER BY revenue_sum DESC
""").df()

It acts as a Swiss‑army knife for data processing: use SQL when convenient, switch to DataFrames without overhead.

Nothing Is Perfect

Pandas boasts the most mature ecosystem (scikit‑learn, Matplotlib, Seaborn) and benefits from the new copy‑on‑write feature in version 3.0, making it suitable for small‑to‑medium data (<1 GB) and exploratory analysis. Polars offers superior multithreaded performance and low memory usage for datasets up to 100 GB, but its ecosystem is smaller and some familiar Pandas operations (e.g., apply) are intentionally harder. DuckDB delivers unparalleled SQL speed and out‑of‑core capability, though it requires SQL proficiency and its DataFrame integration is not as seamless as native Polars.

Which One to Use?

If the dataset fits comfortably in memory (<1 GB), Pandas is a safe choice, especially when leveraging its extensive ecosystem. For larger in‑memory workloads (1 GB–100 GB) where multithreaded speed and memory efficiency matter, Polars is preferable. When the data cannot be loaded into memory or when fast SQL analytics are required, DuckDB is the best fit. Many teams adopt a hybrid approach: DuckDB for ingestion and heavy SQL, Polars for feature engineering, and Pandas for the final machine‑learning and visualization steps.

Conclusion

Pandas remains viable but is no longer the default for serious data work. Polars excels at fast, memory‑efficient DataFrame operations, while DuckDB shatters expectations with SQL performance that outpaces even distributed engines.