DuplexAgent-RSI: Full-Duplex Voice Agents That Talk, Work, and Self-Improve

Overview

Researchers from the Chinese University of Hong Kong (Shenzhen) and Amphion Technology present DuplexAgent-RSI, a full-duplex voice agent system that enables continuous collaboration between real-time speech interaction and background task execution. The system introduces a six-module collaboration control layer and a recursive self-improvement loop (Duplex-Harness-RSI) that automatically tests, diagnoses failures, and applies verified patches to the coordination logic.

Full-Duplex Collaboration Architecture

DuplexAgent separates the real-time interaction model (listening and speaking) from backend reasoning and coding agents. A collaboration control layer coordinates task dispatch, state management, context passing, and result delivery. The article illustrates the challenge with a scenario: a user asks for a Snake game, then cancels and requests Tetris while the first task is still running. The system must invalidate the stale task, adopt the new goal, and deliver results only when the user pauses.

Six Collaboration Modules

The control layer is decomposed into six modules, each with a distinct responsibility:

Policy : decides whether to answer directly, delegate execution, or control an existing task.

Intake : understands the user request and maps it to a task.

Hold : maintains dialogue and progress feedback during task execution.

Worker : selects the appropriate backend agent and passes the task.

Deliver : determines when and how to return results.

Memory : maintains shared context across dialogue and background tasks.

A task ledger records task IDs, versions, statuses, and delivery history, allowing the system to mark superseded tasks as expired and delay delivery until a suitable conversational break.

Recursive Self-Improvement via Duplex-Harness-RSI

To continuously improve collaboration, the team built Duplex-Harness-RSI, a closed loop of: generate tests → run collaboration → record trajectories → diagnose failures → apply targeted fixes → pairwise validation. The test space covers 19 request types and 22 interaction patterns, varying phrasing, speed, language, accent, speaker, and noise while preserving core requirements (e.g., whether an old task should be invalidated). A reasoning model acts as an examiner to select the next test focus; a coding agent acts as a repair engineer to modify the implicated module. Candidate changes are compared against the current version on the same test batch and validated on an independent hold-out set; only changes that yield reliable improvements are kept. Model weights remain frozen; improvements occur in the harness layer's rules, configuration, and execution logic.

Experimental Results

Evaluations on two interaction-model configurations show substantial gains:

Qwen-Audio-3-Realtime : OBQA accuracy rises from 86.3% to 92.1%; MMSU from 64.8% to 82.7%; BFCL average score from 82.0% to 90.8%; FDB-v3 Pass@1 from 39.0% to 49.0%. Interrupt TOR (turn-taking response rate) improves from 89.0% to 97.0%.

Nvidia-VoiceChat : MMSU from 36.1% to 80.5%; BFCL from 53.5% to 75.8%; FDB-v3 Pass@1 from 16.0% to 47.0%. DuplexAgent outperforms the corresponding Qwen-Audio-Agent on all metrics.

The gains stem from the synergy between backend reasoning and the collaboration mechanism, which correctly routes requests, preserves context, and validates task relevance before delivery.

Self-Improvement Trajectory

On the Nvidia-VoiceChat configuration, the initial harness scores 54.6% on a fixed core test set. A one-time batch edit reaches 57.4%; ten sequential edits reach 61.9%. The full Duplex-Harness-RSI loop achieves 85.3% after ten rounds — a 30.7 percentage-point lift. The failure mode "should have delegated but did not" drops from 30.3% to 1.3%. In a separate Qwen-Audio-3-Realtime experiment, eight validated updates raise the score from 47.1% to 84.8%. The process retains repair history, uses pairwise testing, and relies on an independent protection set to ensure each round accumulates reliable collaboration ability.

Future Work

The current recursive self-improvement operates on the collaboration harness while keeping interaction-model weights fixed. The team plans to extend the test simulator, time-aware interaction scenarios, and verifiable collaboration trajectories to Agentic RL, enabling the interaction model itself to learn from full-duplex collaboration failures. This would close the loop from harness-level to model-level recursive improvement.

Paper: https://arxiv.org/abs/2610.11299

Project page: https://amphionteam.github.io/DuplexAgent-RSI-Demo/