How AI Powers Smart Home Workouts on Mobile: Alibaba Sports’ Pose‑Tracking

AI‑Powered Smart Home Workouts

Over the past year, Alibaba Sports’ technology team has explored "endpoint intelligence" for sports and health, launching an AI‑driven workout project that digitizes exercise experiences. Users only need a smartphone and 3‑4 m² of space; the app guides them via voice prompts and starts tracking once the body is fully inside the detection frame.

Technical Foundations

The system runs on Alibaba’s deep‑learning inference engine MNN, performing real‑time pose estimation on the device. It detects 14 key skeletal points (head, shoulders, knees, etc.), connects them to form actions, and analyzes posture, joint angles, and motion trajectories to provide instant feedback and correction.

Real‑time detection of human contours and 14 key points.

Construction of action sequences from connected points for posture and angle analysis.

Action matching, timing, counting, and feedback to improve interaction.

Key Challenges

Deploying intelligent workouts on mobile devices raises several issues:

Ensuring accurate action matching and algorithm design.

Limiting resource consumption (battery drain, device heating).

Improving testing efficiency and coverage across diverse devices and environments.

Solutions for Accuracy and Performance

To boost recognition precision, the team focuses on skeleton‑point stability, selects representative actions for state‑machine modeling, and guarantees sufficient frame rates to cover all states. Performance optimizations include:

Reducing unnecessary data‑format conversions before inference (e.g., direct RGBA conversion).

Choosing lightweight models and appropriate input resolutions for different devices.

Using platform‑specific rendering (Metal on iOS) to lower post‑inference overhead.

Automated Testing Framework

Traditional manual testing proved costly and inconsistent. Alibaba Sports therefore built an AI‑driven automatic testing tool that batch‑processes video samples, simulates real‑world scenarios, extracts skeletal data, evaluates business outcomes, and generates reports. This dramatically reduces labor, improves coverage, and quantifies model accuracy and efficiency.

Business Impact

The AI workout platform now supports dozens of actions (e.g., arm swings, push‑ups, squats, rope skipping) and offers modular AI training courses. Celebrity‑led “star trainer” sessions run 52 weeks a year, encouraging habit formation and boosting user engagement. Continuous expansion of motion libraries and interactive features solidifies Alibaba Sports’ unique smart‑exercise brand.