Flint-40B-A2B: AI-Built MoE Coding Model Hits 125 tok/s on MacBook Pro

Flint-40B-A2B: An AI-Built MoE Coding Model for Local Hardware

Recursive_SI, founded by Tian Yuandong, has released its first model, Flint-40B-A2B, a Mixture-of-Experts (MoE) model with 40 billion total parameters and 2.5 billion activated parameters per token. The model was developed from scratch in under four months by a small team using Recursive's automated research system, Eureka Machine, which participated in nearly every stage of development: data processing, training recipe design, infrastructure, and inference optimization.

Benchmark Performance

On agentic coding benchmarks, Flint outperforms all other MoE models under 50B total parameters. It leads on SWE-Bench Pro v2, Terminal-Bench 2.1, and SWE-Bench Verified, and achieves top scores on IFBench and IFEval. Its coding ability rivals dense models in the 25B–40B parameter range while running more than 5× faster. However, some dense models still score higher on certain coding benchmarks.

Local Inference Speed

Flint's inference engine is optimized for Apple Silicon and NVIDIA RTX GPUs. On a MacBook Pro with M4 Pro chip, it generates over 125 tokens per second; on an NVIDIA RTX Spark with 64 GB memory, it exceeds 150 tokens per second. These speeds are up to 2× faster than comparable models. Independent developers have reported 114–132 tok/s on M5 Max (64 GB) and ~46 tok/s on an RTX 3060 (12 GB VRAM) with 256K context.

Model Architecture Search

Eureka Machine started from a 35B/3B baseline and explored expert configurations and network widths. It predicted that a 40B total / 2.5B active configuration would best balance inference speed and coding capability, which became the final Flint architecture.

Quantization with Offline Distillation

Unlike common practice, the output layer was also quantized to 4 bits (NVFP4). Because the output layer accounts for a large fraction of the activated weights per token, this reduces inference overhead. To recover accuracy, Eureka Machine added an offline distillation step: it pre-sampled responses from a curated core dataset using the 16-bit model as teacher, then trained the quantized model on those responses. This cut training steps by nearly half, and the final NVFP4 model scores within 0.7 points of the 16-bit baseline across benchmarks.

Custom Inference Kernels

Eureka Machine developed specialized kernels for low-bit projection, MoE routing, attention, and prompt processing, tailored to the model structure and hardware. On M4/M5 chips, a large Metal fused kernel merges the entire decoding forward pass into a single kernel, eliminating launch overhead. These kernel optimizations, combined with architecture and quantization choices, compound to deliver the observed speedups.

Open Release and Compatibility

Flint's weights are available on Hugging Face (https://huggingface.co/RecursiveSI/flint) under Apache 2.0. The model works with llama.cpp, Ollama, Unsloth, LM Studio, vLLM, and SGLang. Recursive also open-sourced a lightweight inference engine, rec (https://github.com/recursive-oss/rec), optimized for M4 Pro and M5 Pro/Max with a simple setup process.

Usage Recommendations

Flint supports up to 512K context tokens; Recursive recommends staying within 256K for best results. For complex tasks, break work into smaller subtasks and provide explicit instructions. While Flint is capable, it may require more guidance than large closed-source coding models.

Future Direction

Recursive aims to build recursively self-improving superintelligence by automating the knowledge discovery process. Flint is the starting point: stronger models will make Eureka Machine a better researcher, accelerating the next generation of models at the same cost. The team plans to continue developing more capable local models and the infrastructure to run them, extending this approach to broader AI research and development.