Polars 2.0 Upgrade: Join Row Order, Int128 Promotion, and Streaming Defaults Tested
This article tests Polars 2.0 against 1.43.2, revealing that join row order is no longer stable, Int64+UInt64 operations now return Int128 instead of Float64 for exact precision, and LazyFrame.collect() defaults to the streaming engine—though not all queries gain out-of-core capability.
Polars 2.0 introduces several breaking changes that affect existing data pipelines. The author compares version 1.43.2 with 2.0.0 using a reproducible sales-report workflow and dedicated integer-boundary probes.
Join row order is not guaranteed
The migration guide states that operations like join, group_by, and unpivot do not preserve observed row order. In the example, two CSV files (orders and products) are scanned with explicit schema overrides, left-joined on product_id, a net_cents column is computed, and results are grouped by category and region with an explicit sort. Both versions produce 5,000 joined rows and identical content hashes (
59bdaff3c892c22011795e0dec4acedf23c44b1697b3d0b1973bda14baf2e8fd), confirming content equality. However, when the unsorted join result is indexed with with_row_index("join_position"), the positions of specific order IDs (e.g., O-00001 at row 0 in 1.43.2 vs. row 431 in 2.0.0) differ and can vary across runs. The takeaway: any downstream process that depends on row order must add an explicit sort() on business keys.
Int64 + UInt64 now promotes to Int128 for exact precision
Polars 1.43.2 used Float64 as the supertype for mixed signed/unsigned 64-bit integer arithmetic, causing precision loss for values beyond 2<sup>53</sup>. The article constructs four probes, the most striking being Int64(1) + UInt64(9007199254740993). The exact sum is 9007199254740994. In 1.43.2 the result type is Float64 and the value becomes 9007199254740992.0 (off by 2). In 2.0.0 the result type is Int128 and the value is exactly 9007199254740994. All four probes match the exact integer expectation in 2.0.0, whereas three of four diverged in 1.43.2. The probe code uses a LazyFrame with typed series, adds the columns, and compares the collected value against a Python-computed expected integer. Migration note: even though precision improves, downstream systems that expect a specific schema (e.g., database columns, serialization formats) must be checked for Int128 compatibility.
probe = pl.LazyFrame({
"signed_int64": pl.Series([1], dtype=pl.Int64),
"unsigned_uint64": pl.Series([2**53 + 1], dtype=pl.UInt64),
}).with_columns(
(pl.col("signed_int64") + pl.col("unsigned_uint64")).alias("mixed_sum")
)
expected = 2**53 + 2
print(probe.collect_schema()["mixed_sum"])
print(probe.collect()["mixed_sum"][0] == expected)Default streaming engine does not mean universal out-of-core
LazyFrame.collect()and collect_async() now default to engine="auto", which selects the streaming engine. The release notes list preliminary out-of-core support for sorting, window functions, and many expressions, but explicitly note that join and group_by are not yet covered. The author only verified correctness on the small synthetic dataset; no runtime, peak memory, or disk-spill measurements were taken. Production workloads must be benchmarked independently.
Migration checklist
Pin both Polars versions and the input data.
Compare fully sorted records between versions.
Identify downstream dependencies on row order and add explicit sort() on business keys.
For critical mixed-type expressions, verify both computed values and collect_schema() output.
For high-memory tasks, measure time, peak memory, and disk usage on real data—do not assume streaming solves every scale problem.
The sales summary and integer probes only answer questions for their specific inputs; they do not replace a full production-scale regression suite.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
