LLM Architecture Gallery: A Panoramic View of GPT, Llama, DeepSeek, Qwen, Kimi and More

The LLM Architecture Gallery, built by AI researcher Sebastian Raschka, serves as a standardized reference library that aggregates the architectures of the most prominent large‑language models (LLMs). It is hosted on GitHub and a dedicated website, providing a one‑stop platform for model comparison.

On GitHub the project stores model metadata in a models.yml file, which records fields such as architecture diagram path, parameter count, release information, context length, attention type, normalization method, MoE design, license, and KV‑cache details. The website renders this data in an intuitive interface that supports model search and bidirectional comparison.

The gallery covers the full scale spectrum from hundred‑million‑parameter to trillion‑parameter models, including GPT‑2, Llama 3/4, Gemma, DeepSeek V3/V4, Qwen 3/3.5, Kimi K2, the GLM series, Mistral and others, thereby documenting the evolution of LLM technology from 2019 to the present.

Each model card follows a unified visual framework: a standardized diagram, parameter count, context length, attention mechanism, normalization approach, Mixture‑of‑Experts (MoE) design, licensing, and KV‑cache information are displayed side by side, making architectural innovations immediately visible.

The gallery identifies three major development trends. First, architectures are moving from dense to sparse‑mixed designs; starting with DeepSeek V3, MoE becomes mainstream for trillion‑scale models such as Kimi K2 (1 T) and Llama 4 Maverick (400 B), activating only 3‑10 % of parameters to balance performance and inference efficiency. Second, attention mechanisms have continuously improved—from GPT‑2’s absolute positional encoding and multi‑head attention to RoPE, GQA, MLA, and sliding‑window attention—expanding context windows from 1 024 tokens to the million‑token range. Third, lightweight and heterogeneous parallelism are emerging: models like xLSTM and Mamba‑2 replace self‑attention with recurrent or state‑space modules, while edge‑focused models such as the Gemma 4 E series streamline structures for on‑device deployment.

For the industry, the Gallery breaks the barrier of fragmented documentation and inconsistent standards. Researchers can quickly compare design differences and pinpoint innovation directions; developers can select models based on parameters, licensing, and inference cost; and the open‑source community can reuse the metadata for further development, fostering a thriving ecosystem of open‑weight LLMs.