NLP Model Interpretability: White-box and Black-box Methods and Business Applications

Deep learning is often regarded as a black-box model. Although it can achieve excellent performance on various tasks, understanding how it makes decisions remains challenging. This article provides a comprehensive overview of interpretability methods in the NLP field and discusses practical applications in business scenarios.

The article categorizes interpretability approaches into two types: white-box methods and black-box methods. White-box methods require access to model internals to understand the reasoning process, while black-box methods infer model behavior through input-output relationships without requiring internal parameters.

White-box Methods:

1. Neuron Analysis : Visualizing individual neuron activations to discover their functions. Research by Stanford University (Karpathy & Johnson, 2015) found that different LSTM memory cells serve different purposes, such as detecting quotation marks or sentence length. OpenAI (2017) discovered neurons highly sensitive to sentiment classification.

2. Diagnostic Classifiers : Training simple models (typically linear classifiers) on internal vectors to predict linguistic features. Google's 2019 study on BERT revealed that transformer layers perform traditional NLP pipeline tasks (POS tagging, coreference resolution). Lower-layer tasks are learned earlier, while complex tasks like coreference resolution require higher layers.

3. Attention Mechanisms : Analyzing attention weights to understand model reasoning. Research from Stanford and Facebook (2019) found different attention heads extract different information. However, significant debate exists about whether attention truly explains model decisions (Jain & Wallace vs. Wiegreffe & Pinter).

Black-box Methods:

1. Rationales : Identifying key phrases that support model predictions. The ERASER benchmark (DeYoung et al., 2019) provides datasets with human-annotated rationales across 7 tasks, enabling evaluation of whether models use human-expected information.

2. Adversarial Datasets : Testing model robustness by introducing small perturbations. Niven & Kao (2019) showed that adding "not" to BERT reasoning datasets drastically reduces performance. The 2020 ACL best paper "CheckList" provides a systematic framework for behavioral testing of NLP models.

3. Local Surrogate Models : Using interpretable models (like LIME) to locally approximate black-box model behavior. SHAP is another similar approach. However, these methods have limitations regarding robustness and faithfulness.

The authors suggest that black-box methods are more suitable for business落地 applications due to their lower cost and broader applicability, while white-box methods provide deeper insights into model reasoning but are limited to specific model architectures.