How In-Context Co‑Player Inference and LLM‑Driven Evolution Are Redefining Multi‑Agent RL

Overview

Google released two 2026 papers advancing multi‑agent reinforcement learning (MARL). Paper 1 introduces a mechanism‑design approach called in‑context co‑player inference to obtain robust cooperation. Paper 2 presents AlphaEvolve, an LLM‑guided evolutionary system that automatically discovers new MARL algorithms.

Paper 1: In‑Context Co‑Player Inference

Challenges : (1) Equilibrium selection – multiple Nash equilibria cause independent agents to converge to sub‑optimal outcomes; (2) Environmental non‑stationarity – simultaneous learning makes the environment dynamically change. Existing co‑player awareness methods rely on hard‑coded assumptions or strict separation of meta‑learning and inner‑learning time scales.

Proposed solution : Train agents as sequence models against a diverse distribution of co‑players. Agents infer the optimal response directly from the observed context, eliminating explicit meta‑gradients.

Three‑step causal chain

Diversity induces context‑optimal response – Training against a pool of random “table‑type” opponents enables agents to quickly identify opponent behavior and converge to the best response.

Context learners are exploitable – Freeze the Step‑1 agents (Fixed‑ICL). New agents are trained to exploit Fixed‑ICL’s learning dynamics, learning an extortion strategy that forces Fixed‑ICL into unfair cooperation.

Mutual extortion drives cooperation – Two extortion agents initialized from Step 2 are pitted against each other. Their mutual shaping of each other’s context‑learning dynamics eventually leads to cooperative behavior.

Key findings :

Contextual learning acts as a fast‑time‑scale alternative to explicit meta‑learning.

A mixed training pool of learning agents and table agents is essential; reduced diversity degrades the mechanism.

Exploitation vulnerability becomes a driver for emergent cooperation in social dilemmas.

The authors propose the Predictive Policy Improvement (PPI) algorithm and prove that under perfect world‑model assumptions the predictive equilibrium coincides with a Subjective Embedded Equilibrium.

Paper 2: AlphaEvolve – LLM‑Guided Algorithm Discovery

Designing MARL algorithms traditionally relies on human intuition to navigate a vast design space. AlphaEvolve combines the code‑generation capability of large language models (Gemini 2.5 Pro) with the rigorous selection pressure of evolutionary algorithms to automatically discover effective MARL algorithms.

AlphaEvolve framework

Loop:
  1. Select parent algorithms based on fitness.
  2. Use an LLM to propose semantically meaningful code modifications.
  3. Automatically evaluate candidate algorithms on multi‑agent games.
  4. Add successful candidates to the population.

Discovery 1: VAD‑CFR (Volatility‑Adaptive Discounted CFR)

AlphaEvolve uncovered a CFR variant with three non‑intuitive mechanisms:

Volatility‑adaptive discount : An exponential‑weighted moving average (EWMA) of instantaneous regret magnitude dynamically adjusts the discount factor, replacing the fixed discount used in DCFR.

Asymmetric instantaneous boost : Positive instantaneous regret is multiplied by 1.1, while negative regret remains unchanged.

Hard warm‑start + regret‑magnitude weighting : Strategy averaging starts after 500 iterations and each iteration is weighted by its regret magnitude, unlike standard CFR which averages from iteration 1.

Empirical results on benchmark poker and other games show VAD‑CFR converges fastest and achieves the lowest exploitability.

Key code fragment (simplified):

class RegretAccumulator:
    """Volatility‑Adaptive Discounting & Asymmetric Boosting"""
    def update_accumulate_regret(self, info_state_node, iteration_number, cfr_regrets):
        # 1. Compute volatility and adaptive discount
        inst_mag = max(abs(r) for r in cfr_regrets.values())
        self.ewma = 0.1 * inst_mag + 0.9 * self.ewma
        volatility = min(1.0, self.ewma / 2.0)
        # 2. Asymmetric boost
        r_boosted = r * 1.1 if r > 0 else r
        # 3. Sign‑dependent historical discount
        discount = disc_pos if prev_R >= 0 else disc_neg

Discovery 2: SHOR‑PSRO (Smoothed Hybrid Optimistic Regret PSRO)

This variant introduces a hybrid meta‑solver architecture:

Optimistic Regret Matching (ORM) for stability.

Softmax‑based best pure strategy that biases toward high‑payoff actions via temperature control.

Dynamic annealing schedule : mixing factor λ decays from 0.3 to 0.05 and the diversity reward decays from 0.05 to 0.001 during training; evaluation uses fixed λ = 0.01 and no diversity reward.

Training vs. evaluation asymmetry:

Training returns the averaged policy; evaluation returns the final‑iteration policy.

Internal iteration budget: training runs 1000 + 20 × (pop‑size − 1) steps, evaluation runs 8000 + 50 × (pop‑size − 1) steps.

Across 11 benchmark games (Kuhn Poker, Leduc Poker, Goofspiel, Liar’s Dice, etc.) SHOR‑PSRO outperforms static baselines on 8 games and matches or exceeds state‑of‑the‑art performance.

Comparative Summary

Core problem : Paper 1 studies how cooperation can naturally emerge; Paper 2 tackles automatic discovery of effective MARL algorithms.

Key insight : Contextual learning can replace explicit meta‑learning; LLM‑driven evolution can produce non‑intuitive symbolic algorithms.

Method paradigm : Decentralized MARL with diverse training (Paper 1) vs. evolutionary search guided by LLM code generation (Paper 2).

Validation environments : Iterated Prisoner’s Dilemma (Paper 1); Kuhn Poker, Leduc Poker, Goofspiel, Liar’s Dice (Paper 2).

Full PDFs: https://arxiv.org/pdf/2602.16928 and https://arxiv.org/pdf/2602.16301.