Boost Your Data Exploration with pandas‑profiling: Quick Setup and Customization

For anyone involved in data science, data cleaning and exploratory analysis (EDA) often consume about 80% of the project time, directly affecting data quality and model performance.

Before performing EDA, you need to manually inspect new datasets, understand field meanings, and then conduct basic statistical summaries such as mean, variance, min, max, frequencies, quantiles, and distributions.

In R, the skimr package provides richer exploratory statistics than pandas' describe(). In Python, the pandas‑profiling library offers similar, even more extensive, functionality.

Quick Start

Install the library with pip install pandas-profiling, then import and generate a report using a single line of code:

import pandas as pd
import seaborn as sns
from pandas_profiling import ProfileReport

titanic = sns.load_dataset("Titanic")
ProfileReport(titanic, title = "The EDA of Titanic Dataset")

When used in a Jupyter Notebook, the report renders directly in the notebook cell.

You can also call the method on a DataFrame: df.profile_report() produces the same ProfileReport object, which can be exported with to_file() (remember to include the .html extension) or displayed as widgets using to_widgets() and to_notebook_iframe().

Further Customizing the Report

The default report may contain redundant or insufficient information. pandas‑profiling allows extensive customization, which is stored in a YAML configuration file. The main configurable sections correspond to the report tabs:

vars – adjust statistical metrics displayed for each variable.

missing_diagrams – control visualizations of missing values.

correlations – enable/disable correlation calculations and thresholds.

interactions – visualizations of pairwise variable relationships.

samples – preview the first and last rows (similar to head() and tail()).

Example configuration in Python:

profile_config = {
    "progress_bar": False,
    "sort": "ascending",
    "vars": {
        "num": {"chi_squared_threshold": 0.95},
        "cat": {"n_obs": 10}
    },
    "missing_diagrams": {
        'heatmap': False,
        'dendrogram': False
    }
}
profile = titanic.profile_report(**profile_config)
profile.to_file("titanic-EDA-report.html")

Alternatively, write the same options in a yaml file and pass its path via the config_file argument:

df.profile_report(config_file = "your_file.yml")

This separation keeps the code concise and improves readability.

Final Thoughts

The pandas‑profiling library provides a fast, convenient way to perform EDA, offering richer statistics, missing‑value visualizations, and correlation analyses, which can save a lot of time in the early stages of a project.

However, the generated reports are relatively fixed and may become slow for large datasets. For big data, consider sampling the data or using high‑performance backends such as modin, spark, or dask to accelerate report generation.