How RE2 Boosts FAQ Chatbot Accuracy: A Deep Dive into Text Matching Models

Background

In chatbot solutions there are task‑oriented taskbots and knowledge‑oriented qabots. FAQ‑bots (faqbot) match a user query to a pre‑defined question‑answer pair. Two main technical approaches are text classification and text matching; the latter is better for long‑tail or frequently changing FAQs.

Model Overview

We propose a modular matching framework consisting of five layers: Embedding, Encoder, Interaction, Aggregation, and Prediction. Each layer can be implemented with typical components; the Encoder+Interaction layers are packaged as a reusable Block. Stacking multiple Blocks enables repeated inter‑sentence alignment, improving the model’s ability to capture semantic relations.

Our concrete model, named RE2, uses several such Blocks, each with independent parameters. Within a Block, the Encoder (a two‑layer CNN) produces contextual representations, which are concatenated with the original word vectors and previous alignment outputs before being fused via an Augmented Residual Connection.

Key Components

Augmented Residual Connection

Outputs of the n‑th Block are added to its input after concatenation, allowing gradients to flow through deep stacks.

Alignment Layer

We adopt the Decomposable Attention mechanism (Parikh et al., 2016) to compute inter‑sentence alignment.

Fusion Layer

Inspired by CAFE, we compute three scalar features from concatenation, multiplication, and subtraction using separate fully‑connected networks, concatenate the resulting vectors, and project them.

Prediction Layer

For similarity tasks we use a symmetric scoring function; for entailment or QA we employ a two‑layer feed‑forward network.

Experiments

We evaluate on three NLP tasks: Natural Language Inference (SNLI, MultiNLI, SciTail), Paraphrase Identification (Quora Question Pairs), and Question Answering (WikiQA). Metrics are accuracy for NLI and MAP/MRR for QA.

Implementation uses TensorFlow on Nvidia P100 GPUs, 300‑dim Glove embeddings (fixed), He initialization, GeLU activation, Adam optimizer with warm‑up and exponential decay, and block numbers tuned between 1 and 5.

Ablation Study

Removing encoder output from the alignment layer degrades performance, showing the importance of contextual information.

Omitting residual connections reduces accuracy, confirming their role in deep Block stacks.

Replacing concatenation with a highway network performs worse, indicating simple concatenation is more effective.

Impact on Business

Deployed in Alibaba’s DingXiaoMi FAQ service, the RE2 model improved accuracy for seven major industry categories from below 80 % to 89.5 %, raised the base model accuracy to 84 %, and increased effective click‑through for knowledge recommendation from ~14 % to 19.5 %.

Conclusion and Future Work

We present a compact yet expressive semantic matching framework that achieves state‑of‑the‑art results on public benchmarks without relying on large pretrained models. Future directions include incorporating external knowledge (product, activity data), extending to few‑shot classification, and distilling BERT capabilities into RE2 via a teacher‑student approach.