A 35M-Parameter Model Trained on a Single GPU Claims Best Sub-100M Performance

Recent enthusiasm in the AI community celebrates tiny models that outperform much larger counterparts. Last week, developer slvDev demonstrated a 28.9M‑parameter language model running on an $8 ESP32‑S3 microcontroller with 512KB SRAM, 8MB PSRAM, and 16MB flash, generating text at about 9.5 tokens/s.

Building on this trend, independent researcher Harshal Singh announced BarunLM‑35M, a 35,072,768‑parameter base language model trained on a single NVIDIA H200 GPU. Singh claims it is the best model under 100M parameters, citing an average zero-shot accuracy of 41.01% across nine benchmarks (ARC‑Challenge, ARC‑Easy, BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, SciQ, WinoGrande). This surpasses the 230M‑parameter Liquid AI LFM2.5‑Base and even GPT‑2 and Pythia‑160M, with a bootstrap confidence interval of 0.92–2.71 percentage points.

The model’s architecture consists of 12 layers with a hidden size of 448, seven query heads and one shared key/value head. Its most distinctive feature is a 3:1 alternating pattern of local and global attention: three consecutive layers attend only to the surrounding 256 tokens, while the fourth layer performs a full‑sequence attention, reducing quadratic cost while preserving long‑range information.

A second key innovation is a learnable residual selector placed every four layers. This mechanism chooses which intermediate representations to retain, addressing the limited capacity of small models where early useful features might otherwise be overwritten.

BarunLM‑35M also incorporates grouped query attention, 50% partial RoPE, QK normalization, gated attention output, bounded SwiGLU, and tied input‑output embedding weights. The vocabulary is limited to 16,384 tokens to further shrink the embedding matrix.

Training consumed roughly 57 billion tokens—about 162.5 tokens per parameter—drawn from FineWeb‑Edu, Cosmopedia v2, FineMath‑4+, DCLM, CodeSearchNet Python, and CodeParrot Clean, covering educational text, synthetic data, mathematics, web content, and code. The entire 40‑billion‑token run was completed on a single H200 GPU using the Muon optimizer over 40,690 steps.

The released project includes model weights, training and inference code, and full evaluation results under an Apache 2.0 license. BarunLM‑35M is intended for compact language‑model research, teaching, text‑generation experiments, and local prototype development, not for high‑risk domains such as medical, legal, or financial applications.