Next‑Gen VR Interaction via Micro‑Gesture Recognition: The “MiaoKong Virtual Realm” Demo

Project Overview

The Network Intelligence Research Center demonstrated a next‑generation VR interaction platform, “MiaoKong Virtual Realm,” built on micro‑gesture recognition that converts millimeter‑level finger movements into virtual commands with high precision and low latency.

Experience Projects Presented

Electronic Signature Wall (邮你来签) : Users pinch their thumb and index finger to draw 3D signatures that are projected onto a digital wall and stored as lasting memories of the anniversary.

VR Photo Album (邮苑时光) : By sliding the thumb laterally over the index finger, participants flip through graduation photos and historical images in VR, with a single tap to select a specific picture.

VR Cyber‑Ring Toss (邮点文创) : Leveraging 3D models of the university’s new IP characters generated by a large language model, participants play a ring‑toss game where virtual objects possess realistic physical properties.

Key Technical Contributions

Air‑Writing Handwriting Recognition

Using precise hand‑gesture tracking and motion‑trajectory analysis, the team introduced an “air‑continuous handwriting” method. A machine‑learning model learns 3‑D bare‑hand writing kinematics, extracting pen strokes from continuous trajectories to enable natural, fatigue‑free aerial writing.

High‑Precision Hand Pose Estimation

Combining 2‑D image structure with 3‑D point‑cloud geometry, dynamic convolutions extract visual features from images while a sparse‑anchor “aggregate‑interact‑propagate” paradigm iteratively enhances 3‑D features. This approach reduces irregular data access and achieves accurate, robust, real‑time 3‑D hand pose estimation, winning the ICCV25 HANDS MegoTrack championship.

Fine‑Grained Micro‑Gesture Recognition

The team proposed an online micro‑gesture framework based on hierarchical memory and dynamic queries. A long‑short hierarchical memory matrix stores multi‑granular semantic context, while position‑aware dynamic queries jointly encode gesture location and semantics. By fusing current window data with historical information, the system overcomes information truncation in non‑overlapping sliding windows, reaching 90% accuracy for continuous rapid micro‑motions.

Efficient 3‑D Space Interaction

Building on task operation frequency and multi‑granular micro‑gestures, the framework abstracts gesture granularity from thumb actions on other fingers, forming hierarchical gesture layers. Statistical analysis of VR task frequencies creates frequency‑based task hierarchies. Aligning high‑frequency tasks with high‑granularity gestures improves interaction efficiency by 19% and reduces user fatigue by 67% compared with traditional methods.

Technical Roadmap

The overall pipeline integrates high‑precision hand pose estimation, fine‑grained micro‑gesture detection, and physics‑aware hand‑object interaction. By adding hand‑object contact modeling and collision optimization, the system enhances the realism of grasping, throwing, and other manipulations in VR.

Research Impact

The project’s research spans embodied intelligence, human‑computer interaction, and computer vision. Over the past five years the group has published more than 40 papers in top venues (CVPR, ICCV, ECCV, NIPS, AAAI, etc.), earned a CCF‑A paper count of 10 since 2025, and secured the ICCV25 HANDS MegoTrack championship.

Team Introduction

Led by professors Ren Pengfei and Wang Jingyu, the micro‑gesture team focuses on embodied AI, 3‑D hand reconstruction, action recognition, and dexterous manipulation. Their full‑stack technology matrix—automatic data generation, high‑precision pose estimation, fine‑grained gesture recognition, and physics‑intuitive control—aims to enable natural, efficient XR interaction and human‑level dexterous hand control.