Visual Generation Knowledge Fails to Transfer to Text QA: New Research Bridges Gap

Introduction

A paper titled "Can Visual Generation Expand What Unified Multimodal Models Know?" was posted on arXiv on October 15 by researchers from Tsinghua University and Zhipu AI's Tang Jie team. The study investigates whether knowledge acquired by unified multimodal models during image generation can be accessed via pure text question answering.

Knowledge Transfer Probing (KTP) and MagicWorld Dataset

The authors propose a controlled protocol called Knowledge Transfer Probing (KTP). They construct a fictional world named MagicWorld using non-existent entities, attributes, and values (e.g., a fictional character's jersey number, a flag's color sequence) to prevent models from relying on pre-trained knowledge. Each knowledge unit follows an entity–attribute–value triplet. During training, text prompts only request generating the corresponding object without revealing the target attribute value; the answer exists only in the training images. At test time, the images are removed and the model is asked directly in text. If the model answers correctly, it demonstrates that knowledge from image-generation training has become linguistically accessible.

The dataset covers seven task categories, each with 50 knowledge units, expanded across scenarios and viewpoints to yield 8,552 text-to-image training samples. Text descriptions are checked to avoid direct or indirect leakage of answers, and attribute-value distributions are approximately balanced to reduce bias.

Evaluation measures two dimensions:

Visual Accessibility : whether the generated image correctly reproduces the target knowledge.

Textual Accessibility : whether the model can answer the same knowledge via text without image input.

Comparing the two reveals whether knowledge is unlearned, learned but only visually accessible, or fully transferable.

Experimental Setup

Three unified multimodal models with different architectures are tested: BAGEL , SenseNova-U1 , and Zhipu's internal model GLM-Omni . GLM-Omni is initialized from the GLM-4.1V-Thinking pre-training checkpoint and uses a discrete image tokenizer based on X-Omni, providing both image understanding and generation capabilities for interleaved image-text reasoning.

Three training conditions are compared:

Direct Transfer : text-to-image training provides target knowledge; no text QA supervision for that knowledge.

Text Alignment : a subset of entities receives additional text QA examples to link visual knowledge to language answers.

Interleaved Alignment : the model generates the relevant image, observes it, and then continues reasoning to produce a text answer—all within a single training sequence.

Crucially, entities receiving alignment supervision are disjoint from test entities, ensuring improvements reflect generalization beyond memorized answers.

After training, a Visual Chain-of-Thought (Visual CoT) inference strategy is introduced: the model first generates an image, then conditions its reasoning on that image to answer. This is distinct from interleaved training (which changes training) and allows separate analysis of training vs. inference effects.

Results: Large Visual–Text Accessibility Gap

Under Direct Transfer, GLM-Omni achieves 54.00% visual accessibility but only 14.57% textual accessibility —a gap of 39.43 percentage points. BAGEL and SenseNova-U1 show similar gaps.

Interleaved Alignment Consistently Improves Textual Accessibility

Text Alignment yields mixed results: BAGEL and SenseNova-U1 improve slightly, but GLM-Omni drops from 14.57% to 12.57%. In contrast, Interleaved Alignment raises textual accuracy for all three models . GLM-Omni shows the largest gain, reaching 26.86% average accuracy . Per-task gains are dramatic: shape tasks rise from 12% to 64%, spatial relations from 0% to 40%.

Ablation Confirms Image Generation Supervision Is Essential

An ablation on GLM-Omni keeps the interleaved sequence and text supervision but masks the loss on the image portion. Average accuracy falls from 26.86% to 13.71% , confirming that the image generation objective itself—not just the interleaved format or text supervision—drives the knowledge transfer.

Visual Chain-of-Thought Further Boosts Performance

Applying Visual CoT at inference (generate-then-answer) on top of interleaved training lifts all models further. GLM-Omni reaches 41.14% average accuracy , a 14.28-point gain over direct text answering. The effect is especially striking on numeric tasks: direct text answering yields 0% for all models; with Visual CoT, BAGEL achieves 4%, GLM-Omni 36%, and SenseNova-U1 20%.

Persistent Gap: Even Correctly Generated Images Are Not Fully Linguistically Accessible

Despite improvements, the knowledge gap remains. For knowledge units where the model generates a correct image, textual QA accuracy under interleaved alignment is still below 50% for all three models . This indicates that a substantial portion of visually represented knowledge cannot be reliably retrieved via language.

GLM-Omni Technical Details and Implications for Zhipu's Roadmap

The paper reveals that GLM-Omni is built on the GLM-4.1V-Thinking pre-training checkpoint, uses an X-Omni -based discrete image tokenizer, and is designed for interleaved image-text reasoning. This marks a shift from Zhipu's prior focus on coding, complex software engineering, and long-horizon agent tasks (highlighted in GLM-5.3). Recent external work has treated GLM-5.2/5.3 as language-only models requiring external vision adapters. The emergence of GLM-Omni suggests Zhipu is actively developing native multimodal capabilities, and future releases (e.g., GLM-5.4/5.5) may further integrate generation and reasoning to close the visual–textual knowledge gap.