Master Missing Data in Pandas: Essential Techniques for Data Cleaning

Introduction

During data analysis a lot of time is spent on data preparation; this article discusses tools for handling missing values in pandas.

Missing Data in pandas

pandas represents missing values with NaN for numeric data and None/NaN for object arrays. Descriptive statistics ignore missing values by default.

Key Functions

dropna : remove rows or columns with missing data based on a threshold.

fillna : fill missing values with a constant, dictionary, or interpolation methods such as ffill or bfill.

isnull : return boolean mask indicating missing values.

notnull : inverse of isnull.

Examples

Creating a Series with missing values:

string_data = pd.Series(['aardvark', 'artichoke', np.nan, 'avocado'])

Detect missing values: string_data.isnull() Replace a value with None and check again:

string_data[0] = None
string_data.isnull()

Filtering with dropna on a Series:

data = pd.Series([1, NA, 3.5, NA, 7])
data.dropna()

Filtering rows in a DataFrame:

df = pd.DataFrame([[1., 6.5, 3.], [1., NA, NA], [NA, NA, NA], [NA, 6.5, 3.]])
df.dropna()

Using how='all' to drop rows where all values are NA, and axis=1 to drop columns.

Filling missing values: df.fillna(0) Filling with different values per column: df.fillna({1: 0.5, 2: 0}) In‑place filling: df.fillna(0, inplace=True) Forward fill with limit: df.fillna(method='ffill', limit=2) Filling with the mean of a Series:

data.fillna(data.mean())

Parameters of fillna

value : scalar or dict to fill missing data.

method : interpolation method, default 'ffill'.

axis : axis to fill, default 0.

inplace : modify the object instead of returning a copy.

limit : maximum number of consecutive fills.