How Glint-MVT Powers City‑Scale Multimodal AI: Insights from a Tech VP

Before the DACon conference, Dr. Feng Ziyong, Vice President of Technology at Glint Deep Vision, shared core insights and breakthroughs from his team’s work on deploying multimodal AI in city‑scale security applications.

He highlighted the challenges of applying CLIP‑like universal models to surveillance, where data distribution, object scale, clarity, and Chinese compositional semantics differ from internet‑scale training data, and proposed a pragmatic path: “strengthen single modality first, then align multimodality.”

The self‑developed visual foundation model Glint‑MVT uses an interval Softmax loss and millions of virtual classes to build a highly discriminative visual representation base, supplying high‑quality image embeddings for downstream multimodal models such as RWKV‑CLIP and UniME.

To mitigate the scarcity of high‑quality image‑text pairs, the team introduced RealSyn data construction and ALIP/CLIP‑CID data‑purification techniques. These methods raised compositional understanding by about 30% and retrieval accuracy by roughly 35% in vertical scenarios like security.

RealSyn leverages abundant non‑paired multimodal documents, creates a curated image and sentence pool, and matches each image with multiple semantically related captions from different contexts, thereby improving robustness and generalization.

ALIP generates more compatible synthetic captions, while CLIP‑CID removes redundant data, cutting training costs by 50‑60% without sacrificing performance.

Noise‑filtering mechanisms in RealSyn include single‑modality quality filtering, perceptual and semantic redundancy removal, and cross‑modality similarity filtering with semantic balanced sampling.

The DeGLA approach enhances compositional understanding via self‑distillation, achieving a 30% boost on a million‑scale benchmark while preserving the model’s general capability.

In the real‑time system handling millions of camera streams, a lightweight image encoder extracts features stored in a custom vector database, enabling billion‑scale feature retrieval within seconds; text queries are encoded by a larger text encoder and matched against this database.

Looking forward, Dr. Feng stresses that continuous iteration of data and models is essential, but the key to balancing effectiveness and efficiency lies in efficiently integrating existing foundation models through distillation and transfer learning.