How OpenAI’s Circuit Sparsity Makes Large Language Model Reasoning Transparent

Circuit Sparsity model

OpenAI released a 0.4 billion‑parameter language model named Circuit Sparsity in December 2025. The model applies a circuit‑sparsity technique that zeros out 99.9 % of the weights, yielding a sparse computational architecture that can be inspected layer by layer, breaking the opacity of traditional Transformers.

Training techniques

Dynamic forced sparsity : at every training step a dynamic pruning operation retains only the smallest fraction of weights with the largest absolute values (e.g., the top 0.1 %) and forces the remaining weights to zero, compelling the network to learn under an extremely minimal connection budget from the start.

Activation sparsity : activation functions are inserted at key positions such as the attention module, driving neuron outputs toward a binary “either‑or” state and forming clear information channels within the sparse network.

Custom components : RMSNorm replaces LayerNorm to preserve sparsity, and a Bigram lookup table handles simple token prediction, allowing the main network to focus on more complex logic.

Training with these methods produces emergent, function‑specific circuits. Researchers can identify neurons that specialize in detecting a single quote or acting as a logical counter, and the number of active nodes required for a task drops dramatically compared with dense models. A companion “bridge network” maps explanations extracted from the sparse circuits back onto high‑performance dense models such as GPT‑4, providing a tool for analyzing existing large models.

Additional models released on the same platform

HY‑MT1.5‑1.8B multilingual neural machine translation model

Developed by Tencent’s Mixed‑Mode team, this model has 1.8 billion parameters and supports translation among 33 languages plus 5 dialects. It achieves translation quality comparable to a 7 billion‑parameter model while using only one‑third of the parameters. The model supports quantized deployment and integrates with the HuggingFace ecosystem.

Demo URL: https://go.hyper.ai/I0pdR

AWPortrait‑Z LoRA‑based portrait‑enhancement model

The model is a LoRA plug‑in for mainstream text‑to‑image diffusion models. Without retraining the base diffusion model, it significantly improves realism and photographic quality of generated faces by refining facial structure, skin texture, and lighting.

Demo URL: https://go.hyper.ai/wRjIp

Granite‑4.0‑h‑small instruction‑fine‑tuned model

IBM’s 3.2 billion‑parameter long‑context model is built by fine‑tuning a base model with a mixture of open‑source and synthetic data. It employs supervised fine‑tuning, reinforcement learning from human feedback, and model‑merging techniques. The resulting model exhibits strong instruction following, tool‑calling abilities, and is optimized for enterprise‑level multilingual dialogue and code tasks.

Demo URL: https://go.hyper.ai/1HhB9