Answer-Driven Visual State Estimator for Goal-Oriented Visual Dialogue

Goal-oriented visual dialogue is a newly emerging task at the intersection of vision and language. It requires a machine to achieve a specific visual goal through multi‑turn conversations, which has both research significance and practical value.

Recently, Professor Wang Xiaojie’s team from Beijing University of Posts and Telecommunications, in collaboration with Meituan AI’s NLP Center, had their paper “Answer‑Driven Visual State Estimator for Goal‑Oriented Visual Dialogue” accepted at the top ACM MM2020 conference.

The paper identifies two major shortcomings in existing multimodal dialogue systems:

Language encoding treats the answer as a trivial token appended to the question, failing to distinguish different answers (e.g., Yes vs. No), even though the answer heavily influences subsequent visual attention and dialogue direction.

Visual encoding is often static throughout the dialogue, either concatenated with the dynamic language encoding or guided by a simple QA‑based attention, which cannot adapt to different answers.

To address these issues, the authors propose the Answer‑Driven Visual State Estimator (ADVSE), which consists of:

Answer‑Driven Focusing Attention (ADFA) : a gating mechanism that polarizes the attention guided by the current question and then reverses or maintains it based on the specific answer, thereby emphasizing the answer’s impact on dialogue state.

Conditional Visual Information Fusion (CVIF) : a module that adaptively fuses global image information with object‑level difference cues under the guidance of the current QA pair, producing an estimated visual state.

ADVSE is applied to the public GuessWhat?! dataset for both question generation (QGen) and guessing (Guesser). By integrating ADVSE with a hierarchical dialogue‑history encoder, the model achieves state‑of‑the‑art performance on both tasks, surpassing previous methods in success rate and error rate.

The code for ADVSE‑GuessWhat will be released at https://github.com/zipengxuc/ADVSE-GuessWhat .

In summary, the proposed ADVSE highlights the crucial role of answers in visual dialogue by introducing answer‑driven attention updating and conditional visual fusion, leading to superior results on goal‑oriented visual dialogue benchmarks.