VoiceMem: Streaming Dual-Brain Memory Architecture for Real-Time AI Interaction

Current real-time voice assistants like GPT-4o lack persistent memory, causing trust erosion and an inability to understand user emotions. VoiceMem addresses this with a streaming dual-brain architecture designed for multimodal companions.

Why Existing Memory Systems Fail for Voice

Systems such as Mem0, Zep, and MemOS are unsuitable for real-time voice interaction for two reasons:

Latency mismatch: Voice interaction allows ~400ms from user speech end to AI response; existing systems need ~2000ms for retrieval and return hundreds of memories that exceed the voice model's context capacity.

Emotion blindness: They store only semantic facts (what was said, who was mentioned) but ignore paralinguistic cues — tone, pauses, emotional patterns — that constitute half of human communication.

Streaming Dual-Brain Architecture

Left Brain: Informative Memory with Emergent Clustering

The left brain uses a two-layer graph: an associative layer and a retrieval layer (compatible with Mem0/Zep). Nodes represent entities, people, and knowledge. Its core emergence mechanism adapts to long-term accumulation by building internal subgraphs based on node access frequency, then using an LLM to evaluate importance, necessity, and relevance for optimal clustering.

On the LoCoMo long-conversation benchmark, the left brain achieves 91.2% accuracy with only 5 retrieved memories, while Mem0 scores 61.68% even with hundreds of memories — a 95% cost reduction.

Right Brain: Emotional Memory with Long-Short Term Attribution

The right brain models the user's stable personality ( Identity nodes ) and binds emotions and preferences to specific left-brain memories via Joint nodes . It runs a continuous loop: real-time emotion capture in short-term dialogue, plus periodic long-term review to attribute recurring emotions to stable traits.

Ablation shows the right brain lifts emotion/personality question accuracy from ~10% to >70%. Removing it also degrades factual recall, proving emotional context carries unique information.

Streaming Retrieval Pipeline: 134ms End-to-End

VoiceMem hides retrieval inside the standard VAD silence window (300ms) by splitting the process into four parallel stages:

Listening — concurrent ASR and entity/schema matching.

Speech tail (0–200ms) — graph expansion while user finishes speaking.

Anticipation (200–300ms) — predictive query formulation.

Searching (300–340ms) — final backend graph search.

Total dual-brain retrieval completes in 134ms , adding no perceptible latency. The streaming code mirrors VAD interfaces for drop-in integration.

Real-World Evaluation

Beyond academic benchmarks, the team built a life-like evaluation suite covering information, personality, emotion attribution, paralinguistics, and environmental sound. VoiceMem is already integrated as an optional memory backend in Qwen Audio Agent, enabling voice assistants to remember not just text but speaker identity, sound events, and music.

Future Directions

Expand multimodal memory boundaries toward complete "Moments".

Evolve long-term dual-brain dynamics (forgetting, strengthening, abstraction, reconsolidation).

Harden deployment: lower inference cost, multi-process/multi-agent concurrency, and long-running stability for voice companions, embodied AI, smart hardware, and real-time agents.

Memory is not merely a feature of AGI Being; it is the prerequisite for continuous self and consciousness. VoiceMem aims to deliver a plug-and-play, stable, real-time, and continually evolving memory infrastructure for the real-time interaction domain.

First author: Xie Zhifei (PhD, NTU LV-Lab, lead of Mini-Omni, Mega-ASR). Corresponding author: Yan Shuicheng (Distinguished Professor, NUS; former Chief Scientist, Sea Group; 136k+ citations; proposer of Network-in-Network). Project page: https://xzf-thu.github.io/VoiceMem, Code: https://github.com/xzf-thu/VoiceMem, Paper: https://arxiv.org/pdf/2608.26005.