Elon Musk’s xAI Open‑Sources Grok‑1: A 314‑Billion‑Parameter MoE Large Language Model

Elon Musk’s xAI announced that its large‑scale model Grok‑1 is now open‑source, providing both the model weights and the network architecture on GitHub.

Grok‑1 is a 314‑billion‑parameter mixture‑of‑experts (MoE) model, the largest open‑source model by parameter count to date. It contains eight experts, with two activated per token, resulting in 86 billion active parameters during inference.

The model was pretrained in October 2023 without any task‑specific fine‑tuning, and it uses a rotary‑embedding of size 6144, a tokenizer vocabulary of 131 072 tokens, and a context window of 8192 tokens.

Architecturally, Grok‑1 adopts an MoE design with 64 transformer layers, each containing a decoder block, multi‑head attention (48 query heads, 8 key/value heads), and a dense feed‑forward block with an expansion factor of 8 and hidden size 32 768.

Note: The MoE implementation is not highly optimized; it was chosen to simplify model‑correctness verification without custom kernels.

The model weights are distributed via a magnet link and total nearly 300 GB; running the model requires machines with substantial GPU memory, with estimates suggesting eight NVIDIA H100 GPUs for 8‑bit quantization.

Unlike many models that rely on Python‑based frameworks, Grok‑1 is implemented in Rust and the JAX deep‑learning library.

Additional technical insights from Stanford researcher Andrew Kean Gao reveal that Grok applies a softmax over all eight experts and selects the top‑2, differing from the approach used by the open‑source MoE model Mixtral.

Grok‑1 is released under the Apache 2.0 license, making it commercially friendly.

The open‑source move follows Musk’s legal action against OpenAI for not open‑sourcing its models, positioning Grok‑1 as a direct challenge in the large‑model competition.

Other Musk ventures were highlighted: Tesla’s end‑to‑end visual FSD V12 rollout, SpaceX’s third Starship launch, and Twitter’s open‑sourced recommendation algorithm.

“This will be an exciting year – fasten your seatbelts,” said a new xAI employee.

Reference links: GitHub repository and xAI blog post .