Survey of Machine Learning Model Interpretability Techniques

Model interpretability has become a hot research topic in recent years because practitioners not only care about model performance but also want to understand why a model makes certain predictions, which helps improve features, debug models, and build trust.

The article explains that interpretability means the degree to which humans can understand the reasoning behind a model's decisions. It is crucial during model development for debugging and feature engineering, and during deployment for explaining results to stakeholders, especially in high‑risk domains such as finance and healthcare.

Key evaluation aspects include intrinsic vs. post‑hoc interpretability, model‑specific vs. model‑agnostic methods, and global vs. local explanations. Common metrics such as confusion matrix, precision, recall, specificity, lift, ROC/PR/KS curves, cumulative gains, learning curves, and bias‑variance analysis are also discussed.

Permutation Importance measures the drop in model performance when a feature column is shuffled. Example code: from xgboost import XGBClassifier from sklearn.model_selection import train_test_split import eli5 from eli5.sklearn import PermutationImportance path = './census_income_dataset.csv' data = pd.read_csv(path) X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, random_state=400) model = XGBClassifier(...).fit(X_train, y_train) perm = PermutationImportance(model, random_state=1).fit(X_test, y_test) eli5.show_weights(perm, feature_names=X_test.columns.tolist())

Partial Dependence Plot (PDP) visualizes the marginal effect of one or two features on the predicted outcome. Example code: from pdpbox import pdp feature = 'age' pdp_goals = pdp.pdp_isolate(model, X_train, df.columns, feature) pdp.pdp_plot(pdp_goals, feature) plt.show()

Individual Conditional Expectation (ICE) extends PDP by showing the effect of a feature on each individual instance, revealing heterogeneity and interactions.

LIME (Local Interpretable Model‑agnostic Explanations) builds a simple interpretable model around a specific prediction by sampling perturbed instances and fitting a linear model. Example code: explainer = lime.lime_tabular.LimeTabularExplainer(X_train, feature_names=features_name, class_names=['0','1'], categorical_features=data_cat_features, categorical_names=cat_columns, kernel_width=3) predict_fn = lambda x: model.predict_proba(x).astype(float) exp = explainer.explain_instance(X_test[2], predict_fn, num_features=6) exp.show_in_notebook(show_all=False)

SHAP (Shapley Additive Explanations) assigns each feature a contribution value based on cooperative game theory, handling feature interactions and multicollinearity. Example code: import shap explainer = shap.TreeExplainer(model) shap_values = explainer.shap_values(X_test) shap.summary_plot(shap_values[1], X_test) shap.force_plot(explainer.expected_value[1], shap_values[1][row], X_test.iloc[row])

Other advanced methods mentioned include RETAIN , a two‑level attention RNN for sequential data that provides interpretable visit‑ and variable‑level importance, and LRP (Layer‑wise Relevance Propagation) for deep neural networks, which back‑propagates relevance scores to input pixels.

The article concludes with a list of references and resources for further reading on interpretable machine learning.