Skip to content
Validate Dirty CSV Data Before Charts: Pandera Data Contracts for Pandas & Polars
Fundamentals 14 min read

Validate Dirty CSV Data Before Charts: Pandera Data Contracts for Pandas & Polars

The article demonstrates using Pandera to define data contracts for a 1,200-row dirty sales CSV, validate columns, types, ranges, and uniqueness, isolate 32 invalid rows, and visualize clean data with Plotly, while noting Polars integration caveats and when to adopt Pandera in production pipelines.

Data STUDIO
Data STUDIO
Data STUDIO

Data analysis often produces charts that look fine but are built on flawed data: quantities mixed with text, impossible dates, discounts over 100%. Pandas reads the file without error, but downstream aggregations become unreliable. The solution is to define a data contract up front — specifying required columns, types, uniqueness, value ranges, and allowed categories — and enforce it before any analysis.

01 Define the Contract for a Valid Order Table

The demo uses monthly_sales_dirty.csv (1,200 rows, Jan–Oct 2026) with a companion known_issues.csv listing 28 planted defects: text in quantity, missing region, negative unit price, discount >1, invalid dates, duplicate order IDs, and out-of-vocabulary channels. The table has eight columns: order_id, order_date, region, product, quantity, unit_price, discount_pct, channel. Before computing net sales, the contract requires: no missing columns; quantity as positive integer; unit_price > 0; discount between 0 and 1; unique order_id; region, product, channel from business dictionaries.

02 Read CSV as Raw Strings, Then Parse Types

Install dependencies: python -m pip install pandas pandera plotly matplotlib. Read the CSV with dtype="string" and keep_default_na=False to preserve original values (including empty strings) in a raw copy. This avoids Pandas' automatic type inference. Then strip whitespace, replace empty strings with pd.NA, and parse each column: quantity to nullable Int64, unit_price and discount_pct to float, order_date to datetime — all with errors="coerce" so unparsable values become NA for later schema reporting. The original raw DataFrame retains the exact cell contents for forensic lookup.

03 Declare Columns, Ranges, and Uniqueness with Pandera

Using pandera.pandas.DataFrameSchema, define each column with type, nullability, uniqueness, and checks:

import pandera.pandas as pa
from pandera import Check

schema = pa.DataFrameSchema(
    {
        "order_id": pa.Column(str, nullable=False, unique=True),
        "order_date": pa.Column(pa.DateTime, nullable=False),
        "region": pa.Column(str, checks=Check.isin(["华东", "华南", "华北", "西南"]), nullable=False),
        "product": pa.Column(str, checks=Check.isin(["鼠标", "键盘", "显示器", "扩展坞"]), nullable=False),
        "quantity": pa.Column(pa.Int64, checks=[Check.gt(0)], nullable=False),
        "unit_price": pa.Column(float, checks=Check.gt(0), nullable=False),
        "discount_pct": pa.Column(float, checks=[Check.ge(0), Check.le(1)], nullable=False),
        "channel": pa.Column(str, checks=Check.isin(["直营网店", "平台店", "线下门店"]), nullable=False),
    },
    strict=True,
    coerce=False,
)
strict=True

treats extra columns as contract violations; coerce=False demands that prior parsing already produced the expected types. Category values should come from a maintained business dictionary, not hard‑coded literals.

04 Collect All Failures at Once, Then Quarantine Bad Rows

Validate with lazy=True to gather every failure instead of stopping at the first. Catch SchemaErrors, extract failure_cases (a DataFrame with column, check, index, etc.), and save the full report. If any failure lacks a row index (e.g., missing‑column errors), abort. Otherwise, collect unique row indexes from failure_cases["index"], quarantine those rows, drop them from the working set, and re‑validate the clean set to guarantee downstream purity. In this sample, 1,168 rows pass, 32 rows are quarantined, and the failure report contains 32 records (the 4 duplicate order IDs each contribute two rows). A Plotly bar chart summarizes failures by field: order‑id uniqueness accounts for 8 failures, the other six fields 4 each.

05 Visualize Only Contract‑Compliant Data

Compute net sales as quantity × unit_price × (1 - discount_pct) on the validated clean DataFrame. Group by month and region, sort chronologically, and plot a line chart with Plotly Express. A second chart groups by product. A donut chart shows the pass/quarantine split (1,168 vs 32 rows). All charts are labeled as synthetic samples.

06 Polars Integration: Not a Drop‑In Replacement

Pandera offers a Polars backend, but the pandera.pandas schema cannot be copied verbatim. Backend support matrices, execution semantics, and error‑report structures differ. Before migrating, consult the official backend support table, verify that existing rules are available in the target version, and test lazy execution, exception collection, and row‑index‑based quarantine logic with real project data.

07 When to Adopt Pandera

Introduce Pandera when a DataFrame repeatedly feeds reports, models, or shared pipelines and its structural rules (columns, types, uniqueness, ranges) are stable. Place the schema at the ingestion gateway so downstream steps consume only validated data, simplifying root‑cause analysis. For one‑off exploratory scripts or fluid business rules, lightweight assertions and sampling may suffice. In production, pair the schema with a maintained business dictionary, pinned library/back‑end versions, and a clear process for storing and handling quarantined records. Pandera answers "does this data match our written rules?" — whether the rules themselves are correct remains a business decision.

References

Pandera stable docs & backend support matrix — pandera.readthedocs.io

Pandera Pandas error reporting — pandera.readthedocs.io

Pandera Polars integration guide — pandera.readthedocs.io

Plotly Python line chart docs — plotly.com

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

CSVdata cleaningdata validationPandasPlotlyPolarsdata contractsPandera
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.