Tsinghua’s TianMou Chip Team Returns to Nature Cover with Brain‑Inspired Complementary Vision Paradigm

Multimodal large models, world models, and embodied agents are constantly pushing AI capabilities, yet a fundamental problem—visual degradation in open‑world environments—remains largely ignored. Motion blur, aliasing, over‑exposure, and flicker distort image structure and cause information loss, and because reliable ground‑truth is unavailable, such data are rarely used for supervised training, creating blind spots in AI perception.

From 0 to 1: A New Perception Entry for Degraded Vision

The 2024 Nature cover work introduced a brain‑inspired complementary vision paradigm based on visual primitives. By decomposing open‑world visual input into primitive representations, the system creates two complementary pathways: a Cognitive Path (COP) that preserves full‑color RGB information, and a Motion Path (AOP) that captures high‑precision ±7 bit, 130 dB dynamic‑range, high‑speed sparse data, including spatial differences (SD) and temporal differences (TD). Each pathway responds differently to degradation—RGB retains color but suffers from blur and over‑exposure, SD preserves edges but lacks motion cues, and TD is highly sensitive to change but may misinterpret flicker as motion. Their combination supplies color, structure, and change evidence, enabling AI systems to make accurate judgments despite varied degradations.

From 1 to 10: Learning to Trust Degraded Signals

Building on the chip, the 2026 Nature Sensors paper presents a two‑stage brain‑inspired learning framework. First, a bottom‑up stage learns self‑supervised representations from degraded data without any perfect‑image supervision, embedding visual priors into intermediate embeddings that predict and verify each other. The second, top‑down stage constructs an internal model (IGFNet) that uses cross‑path attention and memory modules to weight reliable cues and infer missing structures, producing an estimated visual ground‑truth (e‑VGT) in latent space.

From "Seeing" to "Understanding": Feeding Perception into Tasks

In the top‑down stage, e‑VGT serves as a new canvas for downstream models. For monocular depth estimation, the team built the Tianmouc‑MDE dataset from extreme‑scene captures and integrated the pre‑trained IGFNet encoder into depth networks, preserving continuous depth structure despite exposure changes. For video instance segmentation, the Tianmouc‑VIS dataset and a YOLO‑CVS detector were created; even with limited data, e‑VGT‑generated labels significantly improved detection and segmentation under harsh conditions.

Ecosystem and Impact

The paper also releases the open‑source TianMouCV library (github.com/Tianmouc/tianmoucv), which already underpins several top‑conference papers. Notable examples include STGDNet (CVPR 2026) demonstrating blur‑free imaging via simultaneous RGB, SD, and TD capture; CSVO (IROS 2025) achieving stable visual odometry under over‑exposure; and a cascaded bidirectional diffusion model (CBRDM) for high‑quality video reconstruction (ICCV 2025).

Beyond research, the technology is being industrialized by Xijian Technology. Leveraging the RGB+TD+SD fusion architecture, they achieve over 90 % reduction in redundant visual data, a >20× reduction in token‑to‑first‑frame latency, >10× increase in token‑processing speed, and an order‑of‑magnitude drop in model training cost, thereby reshaping the input paradigm for large‑scale vision models.

Overall, the TianMou chip and its complementary vision ecosystem demonstrate how brain‑inspired visual primitives and self‑supervised learning can provide robust, high‑information‑density perception for Physical AI across robotics, autonomous driving, AR/VR, and intelligent manufacturing.