How to Evaluate Machine Learning Model Performance Before Production Deployment

The article explains how to evaluate machine‑learning models before putting them into production, using a case study of employee attrition prediction.

Case: Predict Employee Attrition

The dataset (from a Kaggle competition) contains 1,470 employee records with 35 features covering background, job details, history, and compensation, plus a binary label indicating whether the employee left the company.

Model Performance Overview

Two models are trained: a Random Forest with ROC AUC 0.795 and a Gradient Boosting model with ROC AUC 0.803. While ROC AUC suggests comparable performance, deeper analysis is required.

Beyond Accuracy

Because the target class (attrition) is only 16% of the data, accuracy is misleading; a naive model that predicts everyone will stay would achieve 84% accuracy.

Choosing a Model

The Evidently open‑source library is used to generate side‑by‑side performance dashboards, allowing inspection of confusion matrices, class‑wise metrics, and other visualizations.

comparison_report = Dashboard(rf_merged_test, cat_merged_test, column_mapping = column_mapping, tabs=[ProbClassificationPerformanceTab])
comparison_report.show()

Example 1: Tagging Employees

Integrate the model into an HR system to display a "high‑risk" or "low‑risk" tag for each employee. In this scenario, higher recall is preferred to catch as many potential leavers as possible, even at the cost of some false positives.

Example 2: Proactive Alerts

Use model predictions to send alerts to managers. Adjust the probability threshold (e.g., 0.6, 0.8) to balance precision and recall: a higher threshold reduces false positives but also lowers recall. The trade‑off can be visualized with precision‑recall tables and class‑separation plots.

Example 3: Selective Model Application

Analyze segment‑level performance (e.g., by job level, stock‑option level) to decide where each model works best. Low‑performing segments can be excluded from automated decisions, or additional data can be collected to improve those segments.

Model Understanding

Feature‑level quality tables reveal which employee groups each model predicts well, helping to explain model behavior and guide further data collection or rule‑based adjustments.

Ethical and Practical Considerations

The article highlights data limitations (missing attrition types, lack of performance metrics, no timestamps) and stresses the need to evaluate bias, fairness, and the ethical impact of using such models for individual employee decisions.

References

Dataset: https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset

Evidently library: https://github.com/evidentlyai/evidently

Jupyter notebook: https://github.com/evidentlyai/evidently/blob/main/evidently/examples/ibm_hr_attrition_model_validation.ipynb

Original blog: https://evidentlyai.com/blog/tutorial-2-model-evaluation-hr-attrition