How Contextual Co-Player Inference Enables Robust Multi-Agent Cooperation

Overview

Google released two papers that push forward multi‑agent reinforcement learning (MARL). The first paper proposes contextual co‑player inference to enable robust cooperation without explicit meta‑learning. The second paper introduces AlphaEvolve, an LLM‑driven evolutionary system that automatically discovers new MARL algorithms such as VAD‑CFR and SHOR‑PSRO.

Paper 1: Multi‑agent cooperation through in‑context co‑player inference – Core theme: achieving multi‑agent cooperation via contextual inference (Feb 19 2026).

Paper 2: Discovering Multi‑agent Learning Algorithms with Large Language Models – Core theme: automatic discovery of MARL algorithms using LLMs (Feb 24 2026).

Multi‑Agent Cooperation via Contextual Co‑Player Inference

Robust cooperation among self‑interested agents is a fundamental challenge in MARL. Two major obstacles are:

Equilibrium selection problem: Multiple Nash equilibria cause independently optimized agents to converge to sub‑optimal outcomes (e.g., mutual defection in social dilemmas).

Environmental non‑stationarity: Simultaneous learning of other agents makes the environment dynamically change from a single‑agent perspective.

Existing “co‑player learning awareness” methods rely on hard‑coded assumptions or strict separation of naive learners and meta‑learners.

Core Innovation: Contextual Co‑Player Inference

The paper assumes that training sequence‑model agents against a diverse distribution of co‑players naturally induces optimal contextual responses, eliminating the need for explicit meta‑gradients or time‑scale separation.

Figure 1: Mixed training pool (learning agents + tabular agents) leads RL agents to converge to cooperative behavior; ablations show that training only against learning agents or providing explicit co‑player identifiers causes defection.

Three‑Step Causal Chain of Cooperation

Step 1 – Diversity Induces Contextual Best‑Response

Training agents only against a random pool of tabular opponents enables rapid opponent identification and convergence to the best response within a single episode.

Figure 2A‑B: PPI agents (trained only against tabular opponents) quickly adapt to different fixed strategies during evaluation.

Step 2 – Contextual Learners Are Exploitable

Fixing the Step‑1 agent as a “Fixed‑ICL” and training a new agent to exploit it yields an extortion strategy, where the new agent shapes the Fixed‑ICL’s learning dynamics to obtain higher rewards.

Figure 2C‑D: A newly trained RL agent exploits the Fixed‑ICL, forcing it into an unfair cooperative equilibrium.

Step 3 – Mutual Extortion Drives Cooperation

When two extortion agents are trained against each other, they mutually shape each other’s contextual learning dynamics, eventually converging to cooperative behavior.

Figure 2E‑F: Mutual exploitation leads to cooperation both within a single episode and across training iterations.

Key Conclusions

Contextual learning acts as a fast‑time‑scale primitive, removing the need for explicit meta‑learning loops.

A mixed training pool is essential; lack of diversity causes the mechanism to degrade.

Exploitation vulnerability can serve as a catalyst for cooperation, revealing a new emergence mechanism in social dilemmas.

The paper also introduces the Predictive Policy Improvement (PPI) algorithm and proves that under perfect world‑model assumptions, the predictive equilibrium corresponds to a Subjective Embedded Equilibrium.

AlphaEvolve: Automatic Discovery of Multi‑Agent Learning Algorithms

Designing MARL algorithms has traditionally relied on manual iterative optimization, even though methods like CFR and PSRO have solid theoretical foundations. The most effective variants often depend on human intuition to navigate the vast algorithmic design space.

AlphaEvolve combines the code‑generation capability of a large language model (Gemini 2.5 Pro) with the rigorous selection pressure of evolutionary algorithms to automatically discover new MARL algorithms.

Algorithmic Loop

Loop:
  1. Select parent algorithms based on fitness
  2. Use LLM to propose semantically meaningful code modifications
  3. Automatically evaluate candidates in proxy games
  4. Add effective candidates to the population

Discovery 1 – VAD‑CFR (Volatility‑Adaptive Discounted CFR)

AlphaEvolve uncovered a novel CFR variant with three non‑intuitive mechanisms:

Volatility‑adaptive discounting: An EWMA of instantaneous regret magnitude dynamically adjusts the discount factor, unlike DCFR’s fixed discount.

Asymmetric instantaneous boosting: Positive regret is amplified by 1.1×, while negative regret remains unchanged.

Hard warm‑start + regret‑magnitude weighting: Averaging starts after 500 iterations and is weighted by regret magnitude, whereas standard CFR averages from the first iteration.

Figure 1: VAD‑CFR (purple line) shows the fastest convergence and lowest exploitability on most benchmark games.

Key Code Snippet

class RegretAccumulator:
    """Volatility‑Adaptive Discounting & Asymmetric Boosting"""
    def update_accumulate_regret(self, info_state_node, iteration_number, cfr_regrets):
        # 1. Compute volatility and adaptive discount
        inst_mag = max(abs(r) for r in cfr_regrets.values())
        self.ewma = 0.1 * inst_mag + 0.9 * self.ewma
        volatility = min(1.0, self.ewma / 2.0)
        # 2. Asymmetric boost
        r_boosted = r * 1.1 if r > 0 else r
        # 3. Sign‑dependent historical discount
        discount = disc_pos if prev_R >= 0 else disc_neg

Discovery 2 – SHOR‑PSRO (Smoothed Hybrid Optimistic Regret PSRO)

Core innovations of this PSRO variant include:

Optimistic Regret Matching (ORM): Provides stability during learning.

Softmax‑based smoothed best pure strategy: A temperature‑controlled softmax biases selection toward high‑payoff actions.

Dynamic annealing schedule: Mixture factor λ anneals from 0.3→0.05, and the diversity reward decays from 0.05→0.001 during training.

Figure 2: SHOR‑PSRO (brown line) outperforms static baselines on complex games such as 6‑sided Liar's Dice.

Asymmetric Training/Evaluation Design

Mixture factor λ: 0.3→0.05 (annealed) during training, fixed at 0.01 during evaluation.

Diversity reward: 0.05→0.001 (decay) during training, 0.0 during evaluation.

Returned policy: average policy in training, final‑iteration policy in evaluation.

Internal iterations: 1000 + 20 × (pop‑size‑1) for training, 8000 + 50 × (pop‑size‑1) for evaluation.

Full Game Test Results

Figures 3 and 4 summarize performance across eleven benchmark games. VAD‑CFR dominates 10/11 games, while SHOR‑PSRO dominates 8/11 games, reaching or surpassing state‑of‑the‑art results.

Combined Takeaways

Contextual learning offers a scalable path for foundation‑model‑driven multi‑agent systems.

LLM‑guided evolutionary search can uncover non‑obvious symbolic algorithms, shifting MARL design from manual tuning to automated discovery.

For further details, see the arXiv PDFs:

https://arxiv.org/pdf/2602.16928
Discovering Multiagent Learning Algorithms with Large Language Models

https://arxiv.org/pdf/2602.16301
Multi-agent cooperation through in‑context co‑player inference