Can a Single GPU Yield the Best Sub‑100M Parameter Language Model?

BarunLM‑35M is a 35,072,768‑parameter foundation language model released by independent developer Harshal Singh, who claims it is the best model under 100 M parameters. The model was trained on a single NVIDIA H200 GPU using the Muon optimizer for 40 billion tokens over 40,690 steps.

The model’s architecture consists of 12 layers with a hidden width of 448, employing seven query heads and one shared key‑value head. A distinctive design alternates three consecutive layers of local attention (covering the nearest 256 tokens) with a fourth layer that performs global attention, following a 3∶1 ratio. This hybrid local‑global attention reduces quadratic computation while preserving long‑range information.

A second key innovation is a learnable residual selector placed every four layers. It chooses which intermediate representations to retain, addressing the limited capacity of small models where early useful features might otherwise be overwritten.

Additional architectural components include grouped query attention, 50 % partial RoPE, QK normalization, gated attention output, bounded SwiGLU, and tied input‑output embedding weights. The vocabulary size is limited to 16,384 tokens to further reduce embedding parameters.

BarunLM‑35M was evaluated with LM Evaluation Harness 0.4.12 on nine zero‑shot benchmarks (ARC‑Challenge, ARC‑Easy, BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, SciQ, WinoGrande). It achieved an average accuracy of 41.01%, outperforming the 230 M‑parameter Liquid AI LFM2.5‑Base (approximately 6.55× larger) and larger models such as GPT‑2 and Pythia‑160M. Bootstrap resampling (10 k pairs) yielded a confidence interval for the performance gain of 0.92–2.71 percentage points.

The training corpus comprised roughly 5.7 billion tokens from sources including FineWeb‑Edu, Cosmopedia v2, FineMath‑4+, DCLM, CodeSearchNet Python, and CodeParrot Clean, covering educational text, synthetic data, mathematics, general web content, and code. Each parameter saw about 162.5 tokens on average.

BarunLM‑35M is released under the Apache 2.0 license with open‑source model weights, training and inference code, and full evaluation results (GitHub: https://github.com/harrrshall/barunlm-35m, HuggingFace: https://huggingface.co/harrrshall/BarunLM-35M). The author positions the model for compact language‑model research, teaching, text‑generation experiments, and local prototype development, explicitly discouraging use in high‑risk domains such as medical, legal, or financial applications.

Earlier, a separate effort demonstrated a 28.9 M‑parameter model running on an ESP32‑S3 microcontroller (512 KB SRAM, 8 MB PSRAM, 16 MB flash) at roughly 9.5 tokens per second, highlighting the growing feasibility of deploying small language models on extremely limited hardware.

Overall, the work illustrates that with careful architectural choices and efficient training pipelines, powerful language models can be built and trained within the budget of a single GPU, expanding possibilities for on‑device AI.