Can Multi-Agent Systems Be Built Like LEGO? Introducing Agent Primitives for Modular Reuse

Problem and Motivation

Multi‑agent systems (MAS) are usually built for a single task: researchers must redesign agent roles, prompts, and message ordering for each new problem. This hand‑crafted approach contrasts with deep‑learning components such as convolutions or attention, which are reusable across many tasks.

Agent Primitives

The paper abstracts three high‑frequency computation patterns that recur in many MAS and defines them as reusable primitives.

Review Primitive

Implements a generate‑critic‑revise loop that repeats until the answer stabilises. The same loop can be reused for mathematics, code debugging, or open‑domain QA without redesigning a separate critic and refiner for each task.

Voting & Selection Primitive

Generates multiple candidate solutions in parallel, then compares and aggregates their latent representations to reach a consensus. Unlike traditional voting that operates on decoded text, this primitive works directly on hidden states, preserving richer semantic information.

Planning & Execution Primitive

Separates high‑level planning from low‑level execution. The planner produces a roadmap; the executor carries out sub‑goals while passing intermediate states forward. This separation enables the same planner to be applied to math reasoning, program synthesis, or multi‑turn QA.

Latent Communication via KV Cache

Instead of exchanging information as text, primitives pass latent states through the Transformer’s key‑value (KV) cache. The KV cache, originally used to store attention states for efficient autoregressive generation, is re‑encoded with rotary position encoding (RoPE) so that caches from different stages align correctly.

Experiments comparing natural‑language communication with KV‑cache communication show:

When an auxiliary instruction is inserted in the middle of a long context, instruction‑following accuracy drops to 15.6 % for text‑based exchange but remains 73.3 % with KV‑cache.

Adding irrelevant sentences (noise) reduces text‑based accuracy to 73 % with three noise sentences and 47 % with ten; KV‑cache retains 100 % and 93 % respectively.

Automatic Primitive Selection – LLM Organizer

An LLM‑based Organizer analyses a user query, consults a lightweight Knowledge Pool that records successful past primitive configurations, and selects and orders appropriate primitives. Example selections:

Tasks requiring verification and correction invoke the Review Primitive.

Tasks with multiple plausible paths invoke Voting & Selection.

Long‑range tasks start with Planning & Execution, optionally followed by Review.

Experimental Evaluation

The framework was evaluated on eight benchmarks covering mathematics, code generation, and knowledge QA, using five large‑language‑model backbones. Reported metrics:

Accuracy: average gain of 12.0 %–16.5 % over single‑agent baselines.

Token usage & latency: reduced by approximately 3–4× compared with traditional text‑based MAS.

Relative inference overhead: overall cost of 1.3–1.6× a single agent.

Cross‑backbone stability: performance remains more consistent across different model architectures, lowering dependence on any specific model.

Implications

By treating recurring collaboration patterns as modular latent primitives and communicating through KV caches, MAS can achieve higher accuracy and efficiency without the token and latency overhead of text‑based interaction. The Organizer automates composition, turning MAS design from manual pipeline construction into dynamic assembly of standard building blocks.