Boosting Online Speech Recognition with Improved Latency‑Controlled BLSTM Models

Alibaba algorithm expert Kun Cheng presented the paper “Improving Latency‑Controlled BLSTM Acoustic Models for Online Speech Recognition” at ICASSP 2017.

The study aims to achieve higher speech‑recognition accuracy by employing a Latency‑Controlled BLSTM (LC‑BLSTM) acoustic model instead of the standard BLSTM that processes whole utterances.

Unlike standard BLSTM, LC‑BLSTM updates the network with truncated BPTT using a central chunk and a right‑ward additional chunk; the additional chunk is used only for cell‑state computation, while error propagation occurs only on the central chunk. During training, each update processes a small data segment, passing the cell state forward to the next segment, and resetting the backward‑moving LSTM’s cell state to zero at the start of each segment.

This approach accelerates convergence and preserves BLSTM accuracy under acceptable decoding latency, making it suitable for online speech‑recognition services.

However, decoding incurs higher computational cost because a long right‑ward chunk is often needed; with Nc = 30 and Nr = 30 the computation can be twice that of a traditional BLSTM.

The paper proposes two improved LC‑BLSTM models that reduce decoding computation while maintaining accuracy, allowing a server to handle 1.5–2× more concurrent sessions.

In the first improvement, the right‑ward chunk computation is removed for the forward‑moving LSTM, and for the backward‑moving LSTM the right chunk is replaced by a fully‑connected layer whose averaged output initializes the central chunk. Experiments on the Switchboard dataset show over 40 % decoding‑speed improvement with comparable accuracy.

The second improvement replaces the backward‑moving LSTM with a simple RNN, which further cuts computation; Switchboard results demonstrate more than 60 % speedup with only a slight loss in recognition accuracy.

Additionally, the authors found that the backward‑propagating LSTM contributes less than the forward LSTM, so they substituted it with a simple RNN, further accelerating the model; experiments show over 60 % decoding‑speed increase with minimal accuracy degradation.

The full paper is available for free download at the provided link.