Unlock 3D Human Pose Capture with FrankMocap: A Powerful Open‑Source AI Tool

Introduction

Human pose estimation has become a hot research topic in recent years, with applications ranging from autonomous driving and behavior recognition to virtual avatars and interactive games.

What Is FrankMocap?

FrankMocap is a 3D human pose and shape estimation algorithm jointly proposed by the Hong Kong University of Science and Technology and Facebook AI Research. It can estimate not only the body pose but also the body shape and hand motions from a single monocular video.

The algorithm runs at about 9.5 FPS on a GeForce RTX 2080 GPU, though its computational cost makes real‑time performance challenging.

FrankMocap has recently been open‑sourced; the project is available on GitHub and the accompanying paper is accessible on arXiv.

Technical Overview

FrankMocap uses the SMPL‑X human model. Given a color image, two separate network modules predict hand pose and body pose. A fusion module then combines these predictions into a unified 3D full‑body model.

The overall pipeline is illustrated below:

Installation and Dependencies

To use FrankMocap, install the required third‑party libraries, including Detectron2 for hand capture and PyTorch3D for pose rendering. Anaconda is recommended for environment management.

Usage Examples

Below are command‑line examples for different capture modes.

# Body pose only (with monitor)
python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

# Body pose only (headless server)
xvfb-run -a python -m demo.demo_bodymocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

# Hand pose only (with monitor)
python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output

# Hand pose only (headless server)
xvfb-run -a python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output

# Full‑body pose (with monitor)
python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

# Full‑body pose (headless server)
python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output

Further Reading

Project repository: https://github.com/facebookresearch/frankmocap

Paper (PDF): https://arxiv.org/pdf/2008.08324.pdf

Conclusion

3D full‑body pose capture opens up many creative possibilities. Try FrankMocap to explore its capabilities in your own projects.