How to Build Real-Time Voice Recognition on Mobile with TensorFlow Lite

Introduction

Alibaba's Xianyu product focuses on the secondary circulation of idle items, assets, and time, using cross‑platform technologies (Base Flutter/Weex/Dart) and computer‑vision techniques (TensorFlow Lite) on mobile terminals.

Problem Statement

Server‑side voice recognition suffers from two main issues: high latency under poor network conditions, leading to a poor user experience, and heavy resource consumption when traffic spikes.

Solution Overview

To address these problems, the article proposes implementing voice recognition on the client side using TensorFlow Lite, a lightweight framework (~300 KB) that retains accuracy while reducing model size to about a quarter of the original TensorFlow model.

Mel‑Spectrogram Algorithm

The algorithm extracts audio features based on the human auditory mechanism using the Mel‑frequency cepstral coefficient (MFCC) method.

Short‑Time Fourier Transform (STFT)

STFT converts time‑domain signals to frequency‑domain while preserving temporal information, using a Hamming window to handle the non‑stationary nature of audio signals.

Mel Frequency Scale

The Mel scale maps linear Hertz frequencies to a scale that reflects human perception, making low‑frequency changes more noticeable than high‑frequency ones.

Algorithm Optimizations

To achieve real‑time performance on mobile devices, several optimizations are applied:

Instruction‑set acceleration using ARM NEON, yielding 4–8× speedup.

Algorithmic acceleration by limiting processing to the human voice band (20 Hz–20 kHz), reducing sample rate, applying appropriate windowing, and silent‑segment detection.

Sampling‑rate reduction to a maximum of 32 kHz.

Multithreading to parallelize audio segment processing, typically using four threads.

Model Selection and Training

A Convolutional Neural Network (CNN) is chosen for its strength in image‑based classification. The Inception‑v3 architecture is selected for its high accuracy and efficient factorized convolutions.

Training uses 5,000 human‑voice samples and 5,000 non‑voice samples (animals, noise) as positive and negative examples, respectively, with an additional 1,000 validation samples. The TensorFlow session API handles training and inference.

Model Prediction

Audio files are processed through the mel‑spectrogram pipeline, converted to feature images, and fed into the trained Lite model for inference.

References

TensorFlow Lite documentation: https://www.tensorflow.org/mobile/tflite

MFCC and IMFCC speaker recognition research

FFT and STFT transformation diagrams

ARM instruction set overview

TensorFlow Session API: https://www.tensorflow.org/api_docs/python/tf/Session