How End-to-End Speech Recognition is Transforming AI Voice Applications

Recently, the AISummit Global Artificial Intelligence Technology Conference was held online under the theme “Drive·Innovation·Digital Intelligence.” ZuoYeBang’s chief algorithm expert Song Yang was invited to the conference and served as the producer of the “Intelligent Voice Application and Exploration” forum.

Speakers from ZuoYeBang, ByteDance, Microsoft Asia Research, 58.com, Soul Voice and other industry leaders shared forward‑looking insights on intelligent voice from the perspective of their business practices.

Intelligent voice, the simulation of voice information in human‑machine interaction, is one of the three core AI technologies and one of the earliest AI applications. Song Yang, who first encountered speech recognition in the mid‑1990s, recalled early products such as IBM via voice, which required lengthy speaker‑specific training and careful articulation.

Today, mature speech‑recognition technology can transcribe free‑form dialogue from meetings, calls, and video programs with high accuracy, enabling even three‑year‑old children to interact naturally with smart speakers. Song predicts that as multimodal and pre‑trained large models mature, they will further leverage massive data and achieve breakthroughs in low‑resource scenarios.

At the forum, ZuoYeBang’s voice‑technology team leader Wang Qiang introduced the company’s practice in speech recognition, evaluation, pronunciation correction, and speech synthesis. Their end‑to‑end speech‑recognition system emphasizes data efficiency, eliminating the need for traditional HMM‑GMM/DNN pipelines, decision‑tree clustering, alignment steps, and pronunciation dictionaries. By integrating common end‑to‑end models (CTC, CTC‑CRF, Hybrid CTC/Attention) with language models (ngram, rnnlm, transformer‑lm), the system makes more effective use of audio and text data, and all ZuoYeBang scenarios have now switched to this end‑to‑end approach.

Pronunciation correction, a representative exploration at ZuoYeBang, uses computer‑based assessment to identify and correct students’ pronunciation errors, providing real‑time guidance that can be deployed anytime.

The team also highlighted various “atomic capabilities” such as speaker verification, mixed Chinese‑English recognition, speaker separation, and detailed evaluation dimensions including linking, devoicing, stress, and intonation, which support multiple internal product lines.

ByteDance AI Lab researcher Zhang Jun discussed challenges in meeting intelligence and efficiency, presenting algorithms for end‑to‑end speech recognition and downstream tasks.

Microsoft Asia Research senior researcher Tan Xu presented a high‑quality speech synthesis system, analyzing technical difficulties from design to implementation, evaluating the system, and outlining future work.

The AISummit, organized by 51CTO, gathered senior technology leaders to discuss AI’s industry drivers and frontier innovations, covering topics such as computer vision, natural language processing, algorithms, recommendation systems, machine learning, and smart finance.