Nine Common Data Preprocessing Methods Explained

1. Handling Missing Values

Inspect the dataset first to identify missing entries. Deleting all rows with missing data reduces sample size and can bias results if the missingness is not random. Filling missing values with the mean can distort distributions, especially for skewed data such as salaries where extreme values pull the mean upward.

Delete rows only when missing values are few and appear random.

Impute using mean for symmetric distributions, median for skewed data, or mode for categorical fields, aligning missing values with the majority pattern.

2. Detecting Outliers

Outliers may be errors or genuine extreme events. For example, a sales spike ten times the usual value could be a data entry mistake or a real promotion. Deleting outliers indiscriminately can discard valuable insights.

Common statistical methods:

Z‑score : Flag points beyond three standard deviations; works best for roughly normal data.

IQR : Use inter‑quartile range to identify outliers without assuming a specific distribution.

After detection, consult domain experts to decide whether to correct, delete, or retain the outlier.

3. Handling Duplicate Records

Duplicate rows can arise from logging errors or legitimate repeated transactions. Use df.duplicated() to locate duplicates and df.drop_duplicates() to remove them. Beware of “near duplicates” where records appear different (e.g., address variations) but refer to the same entity, requiring fuzzy matching.

Log data often contain exact duplicates within the same second.

Transaction data may have legitimate repeats (e.g., a customer buying the same item twice) that should not be removed.

4. Standardizing Features

Features measured on different scales (e.g., height in centimeters vs. weight in kilograms) can dominate distance‑based algorithms. Standardization removes unit effects.

Z‑score standardization : Transforms data to mean 0 and standard deviation 1; suitable when the data distribution is regular.

Min‑Max scaling : Scales values to the [0, 1] range; a quick option when no special assumptions are needed.

Apply standardization after outlier treatment, as extreme values can skew the scaling.

5. Data Transformation

Many statistical models assume normality, which raw data often violate. Transformations adjust distributions:

Log transform : Effective for positively skewed data with a few large values.

Square‑root transform : Suitable for count data.

Box‑Cox transform : Flexible, parameter‑driven method for a wide range of shapes.

Remember to apply the same transformation to future data before prediction, and reverse it if interpretability requires.

6. Encoding Categorical Variables

Machine‑learning models require numeric input. Two common encoding schemes:

Label encoding : Assign integer IDs to categories (e.g., "Beijing" = 1). Risk: models may infer ordinal relationships that do not exist.

One‑hot encoding : Create a binary column for each category, eliminating false order assumptions. Drawback: high cardinality leads to many sparse columns, possibly requiring alternative methods like target encoding.

7. Processing Time Data

Datetime fields often appear in mixed formats. First, unify them into a standard representation, then extract useful features:

Basic parts : year, month, day, weekday, quarter.

Business parts : holiday flag, weekend flag.

Derived parts : days since a key date, month‑over‑month differences.

These engineered time features are frequently critical for trend analysis and forecasting.

8. Integrating Multiple Data Sources

Enterprise data resides in disparate systems (CRM, ERP, databases, spreadsheets). Use merge operations to combine tables, choosing appropriate join types:

Inner join : Keep only records present in both tables.

Left join : Preserve all rows from the left table, filling missing matches with nulls.

Key challenges include entity alignment (e.g., matching "customer_id" with "user_id") and resolving conflicting values (e.g., differing phone numbers). Manual merges become error‑prone as sources grow, so systematic integration pipelines are recommended.

9. Extracting Data Subsets

More data is not always better. Remove irrelevant features and rows to reduce computation and improve model clarity.

Select relevant columns : Keep only features directly related to the prediction task.

Select relevant rows : Focus on the target population (e.g., customers with spend > 10,000 CNY).

Ensure that filtering decisions are justified; indiscriminate removal can cause the model to overfit a narrow segment and fail on broader data.

Overall, data preprocessing has no fixed recipe; practitioners must assess data characteristics and analysis goals to choose appropriate techniques.