Multimodal AI Research: Video-Aware Dialog, Dual-Channel Reasoning, and Multimodal Machine Translation

Artificial intelligence aims to endow machines with human‑like cognition, but single‑modality approaches (NLP, CV, ASR) overlook the need for integrated multimodal understanding.

Exploration 1 – Video‑Scene‑Aware Dialog: The task generates dialog responses grounded in video, audio, captions, and conversation history. A GPT‑2‑based multimodal unified pre‑training framework is introduced with three pre‑training tasks (Caption Language Modeling, Video‑Audio Sequence Modeling, Response Language Modeling). Experiments on DSTC7‑AVSD and DSTC8‑AVSD achieve first‑place results, with the model described in a diagram.

Exploration 2 – Dual‑Channel Multi‑Hop Reasoning for Visual Dialog: To mimic human iterative focus on image regions, a dual‑channel architecture alternates between track and locate modules, performing multi‑step reasoning over visual and textual modalities. The method attains state‑of‑the‑art performance on VisDial v0.9 and v1.0.

Exploration 3 – Capsule‑Network‑Based Multimodal Machine Translation: A transformer encoder‑decoder is enhanced with two Dynamic Context‑Guided Capsule Network (DCCN) modules that attend to global and local image features at each decoding step. The approach reaches SOTA results on the Multi30K dataset and is accepted at ACM MM 2020.

Exploration 4 – Graph‑Neural‑Network‑Based Multimodal Machine Translation: Fine‑grained graph construction links words and image objects via intra‑ and inter‑modal edges. Node embeddings from GloVe and a custom visual extractor are fused through intra‑ and inter‑modal modules before decoding. This model also achieves SOTA on Multi30K and is published at ACL 2020.

In conclusion, while current AI systems perform well on closed‑domain tasks, achieving human‑level cognition requires continued multimodal semantic understanding, as demonstrated by these research explorations.