How a Dual‑Way Sign Language Digital Human Transforms Communication for the Deaf

Background

Global statistics show a severe shortage of sign‑language teachers; there are over 70 million hearing‑impaired people worldwide, about 27.8 million in China, with a growing demand for accessible communication tools.

User Profile

Gender ratio: 58% female, 42% male.

Age distribution: 2.8% (0‑6 years), 4.3% (6‑14 years), 33% elderly, 59% working‑age adults.

Education: only 6% of school‑age hearing‑impaired children receive special education.

Income: average monthly income around 4,000 CNY, with limited employment opportunities.

Pain Points

Simple daily communication is cumbersome.

Complex scenarios such as medical visits, utility services, and legal proceedings require professional sign‑language assistance.

Information acquisition from audio‑visual media is difficult due to limited sign‑language content.

Product Design

The product targets both hearing‑impaired users and the hearing public (e.g., service staff). It addresses four main scenarios: simple exchange, deep professional exchange, information retrieval, and meeting participation.

Challenges of Sign‑Language Translation

Data scarcity: few sign‑language teachers and limited existing corpora.

Regional variations: diverse signing styles across regions and communities.

Visual language nature: sign order differs from spoken language and lacks fixed grammar.

Technical Solution

The system is built on cloud‑native technology with a pooled digital‑human rendering service. It supports real‑time sign‑language recognition and synthesis on a single device, both in half‑duplex and full‑duplex modes.

Real‑Time Sign‑Language Recognition Pipeline

Capture video frames at 10 FPS, 360 p, and push them via RTC.

When the digital human is active, the media module forwards frames to the sign‑to‑text module.

The recognition module outputs a cumulative list of sign tokens.

Behavior‑tree logic converts tokens to natural text and caches results.

Final events trigger UI updates and optional TTS playback.

Real‑Time Sign‑Language Synthesis Pipeline

Convert natural text to sign‑language vocabulary.

Generate intermediate key‑pose and blend‑shape data.

Render the digital human in Unity and stream the result via RTC.

Behavior Tree Orchestration

The behavior tree coordinates modules such as media, recognition, synthesis, and UI, enabling actions like interrupting TTS, waiting for stream subscription, and displaying subtitles.

Data Production

3D assets (high‑precision models, textures, rigging) are created before motion capture. Optical motion capture with data gloves records sign gestures, which are then refined and integrated.

A sign‑language dictionary defines roots, compound signs, domain‑specific terms, and sensitive words. Crowdsourcing via Alipay collects annotated sign data.

Implementation Details

Single‑device half‑duplex mode mitigates noise and camera placement issues.

High‑performance streaming reduces latency for real‑time interaction.

Conclusion

The project demonstrates the first bidirectional sign‑language translation system that combines pure‑vision recognition, cloud‑native deployment, and a digital‑human interface, addressing both technical and data‑collection challenges.