Mastering Data Mining: A Deep Dive into CRISP‑DM and SEMMA Methodologies

CRISP‑DM Methodology

CRISP‑DM, originally summarized from projects by NCR, Clementine, OHRA and Daimler‑Benz and promoted by SPSS, divides the data‑mining project lifecycle into six phases: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment (often called “Preparation Work”). The diagram (Figure 1) illustrates these stages, noting that the order can be flexible depending on results and tasks.

The outer loop in the figure represents the iterative nature of data mining, emphasizing that insights from one project can guide the next.

1) Business Understanding: Define project goals and requirements from a business perspective, identify actionable data‑mining problems, and draft an initial plan.

2) Data Understanding: Collect raw data, explore its quality, identify interesting subsets, and form hypotheses about relationships.

3) Data Preparation: Transform and clean raw data to create the dataset needed for mining, often repeating tasks without a fixed order.

4) Modeling: Select and apply appropriate modeling techniques, calibrate parameters, and possibly return to data preparation if required.

5) Evaluation: Assess the model against business objectives and ensure all critical issues have been considered.

6) Deployment (Preparation Work): After the model is built, the user decides how to apply it in the operational environment.

SEMMA Methodology

Developed by SAS, SEMMA (Sample, Explore, Modify, Model, Assess) follows a similar flow to CRISP‑DM. It starts with defining the business problem, assessing the environment, preparing data, then iteratively performing mining steps, and finally deploying and reviewing the model.

1) Sample: Gather and combine data to construct the analysis dataset.

2) Explore: Examine data quality (errors, missing values, inconsistencies) and variable distributions, deciding which variables need cleaning or transformation.

3) Modify: Apply corrections, fill missing values, unify units, and perform transformations or standardization as required for modeling.

4) Model: Choose suitable models based on analysis goals (details omitted here).

5) Assess: Perform in‑sample validation, using metrics such as ROC curves or lift charts to evaluate predictive performance.

This article is excerpted from “Financial Business Algorithm Modeling: Based on Python and SAS”, published with permission.