Multi-Agent Decision Large Models: Challenges, Action Semantic Networks, Permutation Invariance/Equivariance, and Automated Curriculum Learning

Real‑world problems involving multiple cooperating agents—such as multi‑hero coordination in games, multi‑user recommendation, vehicle routing, and cloud resource scheduling—can be modeled as multi‑agent decision processes. Modeling these systems as MMDP or Dec‑POMDP leads to exponential growth of joint state and action spaces, creating severe dimensionality challenges.

The main difficulties for multi‑agent reinforcement learning (MARL) large models are: (1) dimensionality explosion of observation and joint‑action spaces, (2) low sample efficiency of existing RL algorithms, and (3) poor generalization across different tasks and environments.

To address these issues three design priors are proposed. First, an Action Semantic Network partitions the global action space according to semantic categories (e.g., self‑movement vs. inter‑agent attacks) and processes each category with specialized modules, improving efficiency and performance in scenarios like StarCraft and Neural‑MMO.

Second, the concepts of Permutation Invariance and Permutation Equivariance exploit the symmetry among homogeneous agents. By designing networks such as Dynamic Permutation Network (DPN) and Hyper‑Policy Network (HPN) that enforce invariance at the input layer and equivariance at the output layer, redundant information is compressed and the model can handle variable numbers of agents without redesign.

Third, Cross‑Task Automated Curriculum Learning selects the next training task based on difficulty and similarity to the target task. Difficulty is estimated by reward evaluation, while similarity is measured via a Gaussian mixture model fitted to state‑visit distributions obtained from rollouts. The HPN architecture enables seamless policy transfer between tasks, allowing the curriculum to iteratively refine a single model that excels on the final, hardest task.

These architectures are integrated with popular MARL algorithms (QMIX, QPLEX, MAPPO) using a minimal‑modification principle, yielding 100% win rates on hard StarCraft scenarios and state‑of‑the‑art results on SMAC‑V1/V2, MPE, and Google Football. The approach also generalizes to varied entity counts and input dimensions, demonstrating strong transferability.

The entire codebase has been open‑sourced as pymarl3 , extending the widely used pymarl2 framework with support for SMAC‑V1/V2, permutation‑aware networks, and curriculum learning, and achieving SOTA performance across benchmarks.

The presenter, Dr. Hao Xiaotian from Tianjin University, invites collaboration to further explore large‑model reinforcement learning.