AI Sign Language Digital Human: Technology, Challenges, and Development by Baidu Intelligent Cloud

Recently a video of Zhu Guangquan testing a sign‑language livestream host went viral. The host, praised for fast and accurate signing, turned out to be the first AI‑driven sign‑language digital human created by Baidu Intelligent Cloud.

The AI sign‑language host is designed to provide 24‑hour sign‑language service for the 27.8 million Chinese hearing‑impaired users during upcoming ice‑and‑snow events, bridging the information gap for sports broadcasts.

Key questions are raised: how does a sign‑language digital human differ from a regular digital avatar, what are the production challenges, and what is its future potential?

Understanding sign language is essential; it is treated as a “minor language” with regional variations. A universal translation rule must be built, similar to standard spoken language, accounting for differences in hand gestures, word order, and linguistic simplification.

Sign‑language translation is not merely text‑to‑text; it requires coordinated hand gestures, facial expressions, and mouth shapes. For example, the phrase “吃饭了吗？” needs both hand gestures for “吃饭” and facial cues for “了么”.

Many existing sign‑language avatars use “gesture‑Chinese” corpora, ignoring sign‑language‑specific syntax and simplification, resulting in awkward or unintelligible output for hearing‑impaired viewers.

Building a high‑quality sign‑language digital human demands integration of speech, vision, and natural‑language‑processing technologies. Baidu leverages its leading AI capabilities—ASR, visual recognition, and NLP—to create a robust pipeline.

The development follows three core model stages:

① ASR Speech‑Recognition Model : Achieves >98 % accuracy across Mandarin, English, dialects, and rare characters, enabling the avatar to understand rapid speech.

② Sign‑Language Translation Model : Converts recognized text into sign‑language symbols, balancing information density and latency. High‑quality data were collected by annotating over a hundred hearing‑impaired students, resulting in a translation intelligibility of >85 %.

③ Gesture‑Generation Model : Generates precise hand, facial, and lip movements. Baidu uses motion‑capture, 2‑D skeleton to 3‑D avatar conversion, and a 4D scanning dataset to produce over 10 000 facial expressions and accurate lip shapes with >98.5 % accuracy.

To ensure correctness, the team follows the 2019 National Sign Language Dictionary, applies finger‑level motion capture, and conducts expert reviews for each gesture, building a library of nearly 10 000 validated sign actions.

The project involved rapid iteration—six major version updates within two months—and close collaboration among AI engineers, sign‑language linguists, special‑education experts, and hundreds of student annotators.

Beyond technical achievement, the AI sign‑language host aims to provide equitable access to information for the estimated 430 million people worldwide with moderate or severe hearing loss, especially in regions lacking sign‑language teachers.

In December 2021, Baidu launched the Xiling Digital Human Platform, offering a one‑stop solution for virtual hosts, employees, idols, and brand ambassadors across media, entertainment, finance, government, telecom, and retail. The AI sign‑language avatar joins this ecosystem, demonstrating how AI can make complex communication more inclusive.