How Shanghai AI Lab Is Turning Pre‑training Into a Fully Open, Transparent Process

Motivation: From Rumors to Transparency

Recent rapid advances in large‑language‑model (LLM) architecture have been dominated by a few compute‑rich tech giants, leaving academic and industrial researchers dependent on private communications and "rumor" channels. The lack of a public, reproducible platform for recording both successful and failed experiments hampers reliable progress.

ArchSpace: An Open‑Source Platform for Full‑Process Disclosure

The Shanghai AI Lab’s Shushu team announces a new development and open‑source model that publishes not only final model weights but also the entire pre‑training lifecycle. The initiative includes:

Full checkpoints and complete training records that reconstruct a model’s growth from zero to final performance.

Small‑scale exploratory experiments with all single‑point architecture attempts disclosed.

Architecture selection and trade‑off rationale , documenting why a particular design was chosen or abandoned.

End‑to‑end validation pipeline covering pre‑training, instruction fine‑tuning, and reinforcement learning, ensuring each proposal is evaluated on the same metrics.

All resources are hosted publicly: the visual website https://www.archspace.live, a GitHub repository https://github.com/InternLM/archspace (continuously updated with code and experiment logs), Weights & Biases logs at https://wandb.ai/archspace, and model weights on HuggingFace https://huggingface.co/ArchSpace-Collection.

Visualization of Architecture Evolution

ArchSpace visualizes the entire evolution from a baseline decoder‑only Transformer to each subsequent architecture innovation. For a new attention scheme, the site shows step‑by‑step scaling from small experiments to larger models, the concrete data and comparative results at each stage, the next model size and key metrics to be validated, and the decision rationale for merging the change into the main research line.

Community‑Driven Validation Workflow

Community members can submit architecture proposals (e.g., "Next Concept Prediction" or "Depth‑Attention") as GitHub issues. After committee review, proposals are scaled and validated at 1B, 3B, and 8B parameter levels, aligned with the fully open Olmo 3 training pipeline (≈6.2 T tokens covering pre‑training, long‑context continuation, and instruction fine‑tuning). All training and evaluation logs are publicly recorded on W&B.

The review committee comprises experts from academia and industry, including Associate Professor Lin Zhouhan (Shanghai Jiao Tong University), Professor Zhang Qi (Fudan University), Assistant Professor Ji Tao (Fudan University), and researchers from the Shanghai AI Lab.

Proposals are evaluated on three dimensions: innovation value, technical feasibility, and completeness of experimental design. Accepted proposals receive collaborative opportunities such as joint internships.

Current Architecture Proposals in the Pipeline

ArchSpace currently hosts more than ten proposals, including:

Next Concept Prediction (NCP) – Liu et al. (2026) [1] Depth‑Attention – Zeng et al. (2026) [2] SiameseNorm – Li et al. (2026) [3] Artificial Hippocampus Networks – Fang et al. (2025) [4] MorphNorm – GitHub issue [5] Mobius – GitHub issue [6] Each proposal awaits full‑scale training and evaluation under the same open framework.

Vision for a New Research Paradigm

By making every training detail, architectural choice, and experimental outcome publicly traceable, the Shanghai AI Lab aims to replace opaque "black‑box" development with a reproducible knowledge asset that the community can cite, extend, and learn from—turning isolated successes and failures into reusable building blocks for future AI breakthroughs.