How Gesture Recognition Transforms Mobile Gaming with Real‑Time AI Control

Project Overview

With the growing demand for richer entertainment experiences, traditional input methods (keyboard, mouse, touch) limit interaction to a 2D plane. To overcome this, a new human‑computer interaction mode—gesture recognition—was introduced, offering natural and intuitive control without high entry barriers.

The project aims to implement a simple, intuitive gesture‑based interface and promote large‑scale civilian adoption of the technology.

System Architecture

The system follows a layered, modular design consisting of three main modules: a front‑end capture module, an algorithm module, and a communication module.

Capture Module

Implemented on Android devices, it continuously captures images via the camera, buffers them for the algorithm, and monitors device status (e.g., camera offline warnings and auto‑reconnect). For devices without built‑in cameras, an external USB camera driver is provided.

Algorithm Module

Responsible for classifying captured gesture images. The development workflow includes dataset selection, augmentation, model design, training, and saving.

Dataset Selection and Augmentation

Two datasets were used:

Five gesture classes from the NUS Hand Posture Dataset II (200 + 75 × 5 = 1375 images).

A self‑collected dataset for TV‑screen demonstrations (5 classes, 540 images) better matching game control directions.

To avoid over‑fitting, the NUS dataset was augmented and split as follows:

250 images (balanced across noise and classes) were reserved for testing.

The remaining 7000 images formed the training set via random cropping (0.8–0.9×) and rotation (±10°).

Images were padded to squares, then z‑score normalized before feeding into the network.

Model Design and Training

Given the limited resources of Android devices, the lightweight MobileNet architecture was chosen and built with PaddlePaddle. Multiple variants were trained (input size 120×120×3) and evaluated for latency and accuracy. The best‑performing model (variant 3) was selected.

Communication Module

After inference, the module sends corresponding control commands to the operating system. Inter‑process communication is achieved by writing shell commands directly to Android, requiring root privileges.

Android Inference

Paddle Lite was compiled for Android (using NDK and CMake) to produce libpaddle-mobile.so. Captured images undergo the same preprocessing as during training, then the saved model is loaded via the compiled library for real‑time inference.

Debugging information—including camera preview, recognition results, and latency—is displayed in a floating window.

Demonstrations

The system controls classic games such as Snake, Temple Run, and Subway Surfers on phones, tablets, and embedded boards (with TV casting). GIFs and screenshots illustrate the gesture‑driven gameplay.

Conclusion

The gesture‑based interaction system enables immersive, hands‑free control of Android applications without requiring wearable devices. Its modular design, lightweight AI model, and cross‑platform deployment demonstrate the potential of AI‑driven HCI to advance game development and related industries.

References

[1] P. K. Pisharady, P. Vadakkepat, A. P. Loh, “Attention Based Detection and Recognition of Hand Postures Against Complex Backgrounds,” *International Journal of Computer Vision*, 2013.

[2] A. G. Howard et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” *arXiv preprint*, 2017.