How Brain‑Inspired Complementary Vision Chips Are Redefining AI Perception in Open‑World Environments

Visual degradation—blur, over‑exposure, flicker, and frequency aliasing—poses a major blind spot for AI systems trained on supervised data, because reliable ground‑truth images are unavailable in such conditions. The authors argue that obtaining high‑quality visual information under limited hardware resources is essential for Physical AI.

Building on the 2024 Nature‑cover work that introduced a brain‑inspired complementary vision paradigm, the 2026 Nature Sensors paper presents a system‑level advance: a self‑supervised learning framework that learns from degraded data without any true‑value labels. The framework first constructs visual primitives (RGB, spatial‑difference (SD), and temporal‑difference (TD)) from the TianMouChip, then uses a two‑stage learning process to build an internal model called IGFNet.

In the bottom‑up stage, visual primitives are embedded into intermediate representations that capture complementary cues: RGB preserves color, SD retains edges, and TD captures rapid motion with ±7 bit precision and 130 dB dynamic range. These representations are combined via cross‑channel attention and a memory module, producing a robust internal model that can predict an estimated visual ground truth (e‑VGT) even when the input signals conflict.

The top‑down stage injects the e‑VGT into downstream tasks. For monocular depth estimation, the team created the Tianmouc‑MDE dataset using e‑VGT as pseudo‑labels, and integrating the IGFNet encoder into a depth network preserved continuous depth structure despite extreme exposure changes. For video instance segmentation, the Tianmouc‑VIS dataset and a YOLO‑CVS model leveraged e‑VGT‑generated annotations, yielding significant performance gains in harsh lighting and motion conditions.

Extensive evaluations on real‑world extreme‑scene datasets collected with the TianMouChip demonstrate that the complementary sensor can capture RGB, SD, and TD simultaneously in a single exposure, reducing bandwidth by over 90 % while cutting token‑to‑first‑frame latency (TTFT) by 20× and increasing token‑processing speed (TPS) by 10×. Training cost is lowered by an order of magnitude, enabling more efficient large‑model vision pipelines.

Beyond robotics, the sparse TD/SD signals support low‑power eye‑movement tracking for AR/VR wearables, and the open‑source TianMouCV library (github.com/Tianmouc/tianmoucv) has already powered multiple top‑conference papers (CVPR 2026, IROS 2025, ICCV 2025). The authors conclude that visual primitives constitute a new visual token paradigm that aligns sensor output directly with transformer‑based models, shifting perception from “capture all pixels then understand” to “capture informative visual tokens and understand”.