Mastering Pandas DataFrame Operations: Alignment, Fill Values, and NaN Handling

Data Alignment

When adding two DataFrames, pandas automatically aligns them by index and column; any mismatched entries are set to NaN (not a number).

First, create two DataFrames:

import numpy as np
import pandas as pd

df1 = pd.DataFrame(np.arange(9).reshape((3,3)), columns=list('abc'), index=['1','2','3'])

df2 = pd.DataFrame(np.arange(12).reshape((4,3)), columns=list('abd'), index=['2','3','4','5'])

Adding them produces a DataFrame where only the overlapping cells contain values; all other positions become NaN. The same behavior applies to subtraction, multiplication, and division, with division also generating inf for divide‑by‑zero cases.

fill_value

To prevent NaNs during arithmetic, pandas offers arithmetic methods that accept a fill_value argument, which substitutes a specified value for missing entries before performing the operation.

Common arithmetic methods include add, sub, mul, div and their reflected counterparts radd, rsub, etc.; the reflected versions reverse the order of operands, enabling expressions like df.rdiv(1) to compute the reciprocal of each element.

Missing‑Value APIs

Before filling missing data, you need to detect it. The isna function returns a boolean DataFrame indicating the location of NaNs.

dropna

The

dropna</p><p>method removes rows or columns containing NaNs. By default it drops any row with a missing value, but you can specify <code>axis=1

to drop columns, and use the how parameter ('all' or 'any') to control the strictness of the drop.

fillna

The fillna function replaces NaNs with a specified value, returning a new DataFrame unless inplace=True is used.

df3.fillna(3, inplace=True)

You can also fill with computed statistics (mean, max, min) or propagate neighboring values using the method parameter: ffill (forward fill) or bfill (backward fill). Forward fill cannot fill the first row because there is no previous value, and backward fill cannot fill the last row.

Summary

The article introduced basic DataFrame arithmetic, highlighted automatic data alignment that can produce NaNs, and demonstrated how to handle missing values using fill_value during operations or the fillna, dropna, and isna APIs afterward. Proper NaN handling is essential for reliable data analysis with pandas.