Skip to content
Active Listening for Machines: Target Speaker Extraction, Visual Attention & EEG Decoding

Active Listening for Machines: Target Speaker Extraction, Visual Attention & EEG Decoding

Professor Li Haizhou's ACM MM 2026 keynote explores how machines can actively listen like humans by combining auditory, visual, and brain signals for target speaker extraction, demonstrating systems that use facial cues, body gestures, EEG decoding, and speech large models to isolate and understand a specific speaker in noisy multi-talker environments.

Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing

From the Chinese Character "听" to Active Machine Listening

Professor Li Haizhou's keynote at ACM Multimedia 2026, titled "Listening the Chinese Way," frames machine listening through the Chinese character "听" (tīng), which comprises "耳" (ear), "目" (eye), and "心" (heart/mind). This decomposition, dating back ~3000 years, illustrates that human listening integrates auditory perception, visual attention, and cognitive understanding. Li connects this to William James's concept of apperception — combining new sensory input with memory and experience to interpret the world. The core thesis: listening is an active, top-down process where the brain directs sensory organs what to attend to, not merely passive sound reception.

The Cocktail Party Problem and Target Speaker Extraction

The "cocktail party problem" — how humans selectively attend to one speaker amid multiple simultaneous talkers — has been studied since the 1950s, initially driven by air traffic control needs. While humans excel at this, conventional speech recognition systems fail in multi-talker scenarios, often requiring users to face the robot directly. The challenge is not just denoising (removing non-speech noise) but target speaker extraction (TSE) : isolating a specific speaker's voice from a mixture of voices. Li's team addresses this by equipping machines with the ability to decide who to listen to.

Multimodal Cues for Speaker Localization

Reference Speech and Memory Bank

Early TSE methods use a reference utterance from the target speaker, feeding both mixture and reference into a speaker encoder to generate a mask. However, robots may not have prior recordings. The team introduced a Memory Bank : on first encounter, the robot captures and stores the speaker's voiceprint, enabling continuous tracking even as the person moves.

Visual Guidance: Face, Body, and Anticipatory Cues

Visual information provides powerful cues. A demo showed two people talking over each other; selecting one person's face in the video allowed the system to extract that person's speech. Visual cues go beyond lip reading: pre-speech facial expressions and body movements predict who will speak next, allowing proactive attention shifting. Even when faces are occluded, body gestures and posture — synchronized temporally with speech across cultures — can guide extraction. The system also accepts text prompts (e.g., "male voice"), reference audio, or photos to identify the target.

Demo: selecting a face in a two-person video extracts that person's speech from the mixture
Demo: selecting a face in a two-person video extracts that person's speech from the mixture

Neuro-Steered Hearing: Decoding Auditory Attention from EEG

Moving beyond external sensors, Li's team explores neuro-steered hearing : using EEG to decode which speaker a listener is attending to, then using that as a control signal for TSE. In a live demo, a participant wearing a 128-electrode EEG cap switched attention between two simultaneous speakers; the system switched extracted audio accordingly with ~200 ms latency, approaching real-time. Li noted that for auditory attention decoding, only ~10 electrodes around the ears are needed, enabling integration into hearing aids or earbuds. This could assist both hearing-impaired individuals and general users in noisy environments.

EEG-based attention switching demo: extracted speech follows the listener's focus
EEG-based attention switching demo: extracted speech follows the listener's focus

Language Knowledge and Speech Large Models for Understanding

Human listeners use linguistic knowledge to fill in missing acoustic information. Inspired by this, the team pursues two directions:

Language-informed extraction: A model (100k–200k parameters, not an LLM) can separate languages in real time. Demo: a TV broadcast with German dialogue overlaid with English narration; the system extracts only the German track.

Language model as training constraint (ELEGANCE): An LM guides audio-visual TSE during training but is removed at inference, adding no runtime cost. The LM ensures extracted speech conforms to linguistic structure, not just acoustic fidelity.

Unifying Extraction and Understanding: TSE-Omni and TS-Encoder

Building toward end-to-end comprehension, the team developed:

TSE-Omni: Unifies audio and visual cues for target speech extraction; uses historical predictions to compensate when visual cues are disrupted.

TS-Encoder (Target-Speaker-Aware Speech Encoding): Embeds target speaker information into speech representations before feeding them to a speech LLM, enabling the LLM to accurately process the intended speaker's content.

On the Libri2Mix clean test set, a baseline Speech LLM achieved a word error rate (WER) of 70.55% . A cascaded WeSep system reduced WER to 12.53% , while TS-Encoder further lowered it to 10.67% , demonstrating that speaker-aware encoding significantly improves downstream recognition.

TS-Encoder architecture: target speaker encoding fed into speech LLM
TS-Encoder architecture: target speaker encoding fed into speech LLM

Robot Demonstration and Future Directions

A live robot demo showed the system identifying people in the scene, highlighting the current focus target with a green bounding box, and processing that person's speech amid multiple simultaneous talkers. The robot uses simple heuristics for initial target selection (e.g., center of frame, loudest voice) and corrects course if the choice proves wrong — an early form of active listening. Audience questions addressed intention recognition (still heuristic-based) and EEG miniaturization for hearing aids (moving from 128 to ~10 electrodes).

Robot demo: green box tracks the attended speaker in a multi-person scene
Robot demo: green box tracks the attended speaker in a multi-person scene

Conclusion

From the tripartite "耳-目-心" of the Chinese character "听" to multimodal fusion of audio, vision, and brain signals, and onward to speech large models and embodied robots, the keynote maps a trajectory from passive perception to active, selective listening with comprehension. For machines, hearing is only the first step; knowing who to listen to and understanding the message are the keys to natural human-machine interaction.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

multimodal learningtarget speaker extractionbrain-computer interfacehuman-robot interactionactive listeningEEG decodingcocktail party problemspeech large models
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.