Why Polars Is the Fastest Python DataFrame Alternative to Pandas

Polars: A High‑Performance DataFrame Library

Polars is a high‑performance DataFrame library for structured data manipulation, often considered a promising alternative to pandas. Its core is written in Rust, but it provides a Python interface. Key features include:

Fast: Polars is built from scratch, tightly coupled with the hardware and has no external dependencies.

I/O: First‑class support for local files, cloud storage, and databases.

Easy to use: Queries are expressed in a natural way; Polars optimizes execution internally.

Out‑of‑core processing: Polars can process data streams without loading everything into memory.

Parallel processing: Workloads are automatically distributed across CPU cores.

Vectorized query engine: Uses Apache Arrow, a columnar format, and SIMD for CPU efficiency.

User guide: https://pola-rs.github.io/polars/user-guide/ API reference: https://pola-rs.github.io/polars/py-polars/html/reference/io.html

Introduction

Polars aims to provide a lightning‑fast DataFrame library with the following characteristics:

Utilizes all available CPU cores.

Optimizes queries to minimize unnecessary work and memory allocation.

Handles datasets much larger than available RAM.

Offers a consistent and predictable API.

Enforces a strict schema (data types are known before query execution).

Polars is written in Rust, giving it C/C++‑level performance and full control over performance‑critical parts of the query engine. The library focuses on:

Reducing redundant copies.

Efficient memory traversal.

Minimizing contention in parallel execution.

Block‑wise data processing.

Reusing memory allocations.

1. Basics

Series & DataFrames

A Series is a one‑dimensional array where all elements share the same data type. The following snippet creates a simple named Series:

import polars as pl
import numpy as np
s = pl.Series("a", [1, 2, 3, 4, 5])
print(s)
print(s.min())
print(s.max())
s = pl.Series("a", ["polar", "bear", "arctic", "polar fox", "polar bear"])
s2 = s.str.replace("polar", "pola")
print(s2)
from datetime import date
start = date(2001, 1, 1)
stop = date(2001, 1, 9)
s = pl.date_range(start, stop, interval="2d", eager=True)
print(s.dt.day())

A DataFrame is a two‑dimensional structure composed of one or more Series. Operations on a DataFrame resemble SQL queries (GROUP BY, JOIN, PIVOT) and support custom functions.

import polars as pl
from datetime import date, datetime

df = pl.DataFrame({
    "integer": [1, 2, 3, 4, 5],
    "date": pl.date_range(date(2022, 1, 1), date(2022, 1, 5), "1d", eager=True),
    "float": [4.0, 5.0, 6.0, 7.0, 8.0],
    "bool": [True, False, True, False, True]
})
print(df)
print(df.head(3))

2. Reading & Writing

Polars provides fast I/O functions:

import polars as pl
from datetime import date

df = pl.DataFrame({
    "integer": [1, 2, 3],
    "date": pl.date_range(date(2022, 1, 1), date(2022, 1, 3), "1d", eager=True),
    "float": [4.0, 5.0, 6.0]
})
print(df)
df.write_csv("output.csv")
df_csv = pl.read_csv("output.csv")
print(df_csv)

3. Expressions

Polars uses a domain‑specific language (DSL) for data transformation. Expressions can be used for selection, filtering, and aggregation.

df = pl.DataFrame({
    "nrs": [1, 2, 3, None, 5],
    "names": ["foo", "ham", "spam", "egg", None],
    "random": np.random.rand(5)
})
# Selection
selected = df.select([
    (pl.col("nrs") + 5).alias("nrs + 5"),
    (pl.col("nrs") - 5).alias("nrs - 5"),
    (pl.col("nrs") * pl.col("random")).alias("nrs * random"),
    (pl.col("nrs") / pl.col("random")).alias("nrs / random")
])
print(selected)
# Logical expressions
logical = df.select([
    (pl.col("nrs") > 1).alias("nrs > 1"),
    (pl.col("random") <= 0.5).alias("random <= .5"),
    (pl.col("nrs") != 1).alias("nrs != 1"),
    (pl.col("nrs") == 1).alias("nrs == 1"),
    ((pl.col("random") <= 0.5) & (pl.col("nrs") > 1)).alias("and_expr"),
    ((pl.col("random") <= 0.5) | (pl.col("nrs") > 1)).alias("or_expr")
])
print(logical)

4. Lazy / Eager API

Polars supports two execution modes:

Eager : Queries are executed immediately.

Lazy : Queries are built as a logical plan and executed only when needed, allowing optimizations and streaming.

# Eager example
import polars as pl

df = pl.read_csv("heart.csv")
filtered = df.filter(pl.col("age") > 5)
agg = filtered.group_by("sex").agg(pl.col("chol").mean())
print(agg)

# Lazy example
q = (
    pl.scan_csv("heart.csv")
    .filter(pl.col("age") > 5)
    .group_by("sex")
    .agg(pl.col("chol").mean())
)
result = q.collect()  # Executes the plan
print(result)

5. Casting (Type Conversion)

Columns can be cast to different data types using cast(). The strict flag controls error handling.

df = pl.DataFrame({
    "integers": [1, 2, 3, 4, 5],
    "floats": [4.0, 5.0, 6.0, 7.0, 8.0]
})
# Cast integers to float32
out = df.select([
    pl.col("integers").cast(pl.Float32).alias("integers_as_floats"),
    pl.col("floats").cast(pl.Int32).alias("floats_as_integers")
])
print(out)

6. Missing Values

Polars provides utilities for handling nulls.

df = pl.DataFrame({"value": [1, None]})
print(df.null_count())

# Fill nulls with a literal
filled = df.with_columns(pl.col("value").fill_null(pl.lit(0)))
print(filled)

# Forward fill
filled_fwd = df.with_columns(pl.col("value").fill_null(strategy="forward"))
print(filled_fwd)

7. Joins

Polars supports various join strategies (inner, left, outer, cross, asof, semi, anti).

customers = pl.DataFrame({"customer_id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]})
orders = pl.DataFrame({"order_id": ["a", "b", "c"], "customer_id": [1, 2, 2], "amount": [100, 200, 300]})
joined = customers.join(orders, on="customer_id", how="inner")
print(joined)

8. Concatenation

DataFrames can be concatenated vertically or horizontally.

df1 = pl.DataFrame({"a": [1], "b": [3]})
df2 = pl.DataFrame({"a": [2], "b": [4]})
vertical = pl.concat([df1, df2], how="vertical")
print(vertical)

h1 = pl.DataFrame({"l1": [1, 2], "l2": [3, 4]})
h2 = pl.DataFrame({"r1": [5, 6], "r2": [7, 8], "r3": [9, 10]})
horizontal = pl.concat([h1, h2], how="horizontal")
print(horizontal)

9. Pivots

df = pl.DataFrame({
    "foo": ["A", "A", "B", "B", "C"],
    "N": [1, 2, 2, 4, 2],
    "bar": ["k", "l", "m", "n", "o"]
})
pivoted = df.pivot(index="foo", columns="bar", values="N", aggregate_function="first")
print(pivoted)

10. Melts

df = pl.DataFrame({
    "A": ["a", "b", "a"],
    "B": [1, 3, 5],
    "C": [10, 11, 12],
    "D": [2, 4, 6]
})
melted = df.melt(id_vars=["A", "B"], value_vars=["C", "D"])
print(melted)