Meta’s Open-Source Llama 4: 2‑Trillion‑Parameter Behemoth Redefines AI

Model Variants

Llama 4 Maverick – 4020 billion total parameters, 170 billion active.

Llama 4 Scout – 1090 billion total parameters, 170 billion active.

Architecture

Mixture‑of‑Experts (MoE) with 128 expert modules; inference activates the two most relevant experts, keeping compute comparable to a small model while scaling parameters.

Native multimodal early‑fusion: trained on >300 trillion tokens spanning text, images, and video; uses a MetaCLIP‑based visual encoder; supports up to eight images per query (48 during pre‑training).

Context window of 10 million tokens.

Training Innovations

FP8 precision training for faster, lower‑cost training.

MetaP automatic hyper‑parameter optimization.

Hard‑prompt filtering discarding ~50 % of “easy” data.

iRoPE (interleaved rotary position encoding) for stable ultra‑long sequence handling.

Benchmark Performance

In the LMSys Arena evaluation Llama 4 Maverick achieved a 1417 ELO score, outperforming GPT‑4o, Gemini 2.0 Flash and matching DeepSeek v3 with roughly half the parameter count. It also ranked above Gemma 3, Mistral 3.1 and Gemini Flash Lite on vision‑language and multilingual benchmarks.

Representative Evaluations

Knowledge test : Prompt “List all countries ending with ‘S’” returned a complete list instantly.

Math solving : Solved a linear system with four equations and three unknowns, correctly identifying consistency and infinite solutions.

Code generation : Given the prompt “Build an HTML snake game”, produced a full HTML implementation that executed correctly.

Image understanding : Counted pencils in a supplied image accurately.

Access

Checkpoints are hosted on Hugging Face at https://huggingface.co/collections/meta-llama/llama-4-67f0c30d9fe03840bc9d0164 and model code at https://github.com/meta-llama/llama-models/tree/main/models/llama4. The official blog post is https://ai.meta.com/blog/llama-4-multimodal-intelligence/.