Breaking Camera Dependence: M4Human Advances Millimeter-Wave Human Perception to New Levels

Research Background

Human‑centric AI systems need to understand full body motion, not just a few keypoints. Camera‑based Human Mesh Reconstruction (HMR) offers rich pose and shape information but raises privacy concerns and suffers from lighting and occlusion issues.

Why Human Perception Can't Rely Solely on Cameras

Visual data directly captures a person's appearance, which can be uncomfortable in medical, elder‑care, or child‑care settings. Moreover, cameras are vulnerable to poor lighting and occlusion, limiting reliability.

Existing Problems

Current RF/mmWave datasets provide only coarse skeleton annotations.

Action diversity is limited, focusing on simple stationary motions.

Raw radar tensors are rarely released; most works use processed point clouds that discard fine‑grained spatial information.

What M4Human Fills

M4Human is a large‑scale multimodal benchmark designed for high‑fidelity RF/mmWave human modeling. It contains 999 sequences, 661 K synchronized frames, 20 participants, and 50 action classes, totaling over 15 hours of data. Unlike prior datasets, it provides RGB, depth, raw radar tensor (RT), and radar point cloud (RPC) together with high‑precision marker‑based MoCap mesh and global trajectory annotations.

Dataset Details

The dataset emphasizes actions that are more representative of real‑world scenarios, including seated and dynamic non‑stationary motions. It also offers both raw RT and processed RPC, enabling researchers to explore end‑to‑end modeling from radar signals to human mesh.

Data Collection and Annotation Credibility

A multimodal capture platform integrates an Intel RealSense RGB‑D camera, a Vayyar imaging mmWave radar, and a Vicon MoCap system. The Vicon system supplies high‑accuracy 3D motion capture. During acquisition, 37 markers are attached to each subject; mesh annotations are generated from MoCap and manually verified to ensure spatial and temporal consistency.

The full pipeline includes sensor setup, spatial calibration, temporal synchronization, mesh generation, and human‑in‑the‑loop quality checks.

Benchmark Design

M4Human defines three split protocols: Random split, Cross‑Subject split, and Cross‑Action split. These evaluate standard performance and the ability to generalize to unseen subjects or action distributions.

RT‑Mesh: Baseline from Raw Radar Tensor

RT‑Mesh is the first baseline that directly regresses human mesh from raw radar tensors. It first performs efficient localization in bird‑eye‑view (BEV) space, then regresses mesh from a local 3D radar tensor. This demonstrates that raw RT can serve as a core input representation rather than merely auxiliary information.

Result 1 – RT Is Not Only Usable but More Stable in Generalization

In radar‑only experiments, RT and RPC achieve comparable performance on the Random split, but RT shows superior stability on Cross‑Subject and Cross‑Action splits. For the ALL protocol, RT‑Mesh attains mean vertex error (MVE) of 90.9 mm (S1), 135.1 mm (S2), and 143.1 mm (S3) with an inference latency of 2.74 ms and ~2.6 GFLOPs, indicating that raw radar tensors retain richer spatial detail than thresholded point clouds.

Result 2 – mmWave Is a Strong Complementary Modality

Radar does not replace cameras; it complements them. Radar‑only performance rivals RGB and approaches depth in many cases. Fusion experiments (Depth + RT, RPC + RT) further improve reconstruction and tracking accuracy. Radar offers two key advantages: better privacy friendliness and robustness to lighting/occlusion, and it excels at root‑trajectory tracking because it is sensitive to moving foreground while ignoring static background.

Conclusion

M4Human pushes RF/mmWave human perception from skeleton‑level pose estimation to full mesh reconstruction, providing a systematic benchmark that supports high‑fidelity modeling, privacy‑preserving sensing, and robust evaluation across diverse scenarios. The dataset and the RT‑Mesh baseline together establish a new research paradigm for embodied AI applications such as smart homes, medical rehabilitation, and human‑robot interaction.