NVIDIA's Advances in Multi‑Role Generative Dialogue Modeling and Synthetic Data‑Driven QA

The presentation introduces two major research directions from NVIDIA: (1) multi‑role generative dialogue modeling and (2) training QA models with synthetic data.

Multi‑role dialogue modeling focuses on generating consistent conversational style by conditioning the model on a target speaker’s historical utterances. Using a GPT‑2‑style autoregressive decoder (named GCC), the approach concatenates token sequences from the target speaker’s past dialogue and the current multi‑speaker context, employing token‑type embeddings (P, R, S, NS) and position embeddings. Experiments on Reddit data show that larger models (up to 3.8 B parameters) produce more natural and style‑consistent responses, and that decoder‑only architectures achieve lower perplexity than Seq2Seq or VAE baselines.

Synthetic QA data pipeline addresses the scarcity of annotated QA pairs by generating answers first (using a BERT‑style answer‑extraction model without a question) and then generating questions conditioned on the document and the generated answer (via a GPT‑2‑like model). A filtering step compares answers from a pretrained BERT‑QA model to retain high‑quality pairs. Experiments on SQuAD 1.1 demonstrate that models fine‑tuned on purely synthetic data can surpass those trained on manually annotated data, and that larger models improve both question generation and filtering quality.

The work also includes detailed analyses of model size impact on the pipeline components, automatic perplexity‑based testing, and human evaluations covering relevance, style consistency, and coherence. Results consistently indicate that scaling model size yields better performance, while answer‑generation quality is less sensitive to model size.

The session concludes with a Q&A covering topics such as knowledge distillation, historical dialogue selection, data‑pattern dependence, and hardware resources (8 × V100 GPUs per node).