Beyond Single LLMs: MoCo, a Multi‑Model Collaboration Framework

Beyond the era of single large language models (LLMs), researchers are exploring model collaboration, where multiple LLMs trained on different data and objectives interact through routing, text exchange, logit arithmetic, or weight merging to form modular, decentralized AI systems.

The University of Washington team, together with collaborators from Stanford and Harvard, released MoCo – a Python framework that supports 26 multi‑model collaboration algorithms spanning four hierarchical levels: API‑level, Text‑level, Logit‑level, and Weight‑level. MoCo standardizes datasets, model specifications, and hardware configurations, allowing systematic evaluation and rapid prototyping of new collaboration strategies.

MoCo currently implements 26 algorithms from the four categories, providing a unified benchmark for comparing approaches and a solid base for designing novel methods.

Key practical features include: a pip‑installable package, a single configuration file to declare participating models, datasets, hardware, and hyper‑parameters, and built‑in support for 25 evaluation datasets covering QA, mathematics, reasoning, code, and safety. The framework runs on any number of GPUs, making it suitable for both large‑scale and resource‑constrained experiments.

Using MoCo, the authors scaled model collaboration systems from 2 to 16 models and observed consistent performance improvements, suggesting new scaling laws for modular AI. They also compared homogeneous ensembles (8 identical LLMs) with heterogeneous ensembles (8 diverse LLMs) and found the latter significantly outperforms the former, highlighting the value of model diversity.

Further experiments showed that collaborative systems can solve problems that no single model can, with an average of 18.5% of previously unsolvable tasks being resolved through interaction, indicating emergent capabilities beyond simple ability aggregation.

Researchers are invited to contribute additional collaboration algorithms to MoCo via its GitHub repository (https://github.com/BunsenFeng/model_collaboration), fostering a community‑driven methodology for modular, decentralized AI development.