Skip to content
Tagged articles

CPU inference

15 articles · Page 1 of 1
Lao Guo's Learning Space
Lao Guo's Learning Space
Nov 16, 2026 · Artificial Intelligence

CPU-Only 284B LLM at 9 Tokens/sec: Why Memory Capacity Now Trumps GPU

The llama-modchip project demonstrates running a 284B parameter MoE model on a 64GB RAM CPU-only machine via aggressive quantization to 56.6GB and kernel optimizations, but achieves only 9 tokens/sec decoding speed, revealing that memory capacity—not GPU—is becoming the key bottleneck for local large language model inference, with practical implications for Mac memory sizing.

CPU inferenceMac hardwareMoE
0 likes · 11 min read
CPU-Only 284B LLM at 9 Tokens/sec: Why Memory Capacity Now Trumps GPU
DataFunSummit
DataFunSummit
Nov 15, 2026 · Artificial Intelligence

284B DeepSeek on 64GB CPU: Quantization Works, 96GB Xeon Fails on AVX-512

The llama-modchip project runs a 284B-parameter DeepSeek-V4-Flash model on a 64GB RAM CPU-only machine via aggressive quantization to 1.71 bits/weight (56.56 GiB), achieving 9–10 tokens/sec, but a 96GB Xeon server fails due to custom GGUF tensor types and missing AVX-512, highlighting hardware-software compatibility gaps.

AVX-512CPU inferenceDeepSeek-V4-Flash
0 likes · 13 min read
284B DeepSeek on 64GB CPU: Quantization Works, 96GB Xeon Fails on AVX-512
DataFunTalk
DataFunTalk
Nov 8, 2026 · Artificial Intelligence

DeepSeek-V4-Flash on 64GB CPU: Why 96GB Xeon Can't Load It

The article explains how llama-modchip runs 284B DeepSeek-V4-Flash on a 64GB RAM CPU-only machine via aggressive quantization (1.71 bits/weight), while a 96GB Xeon server fails due to tensor format mismatch and missing AVX-512. It breaks down inference latency for long contracts (~10 minutes for 20k tokens) and proposes three evaluation metrics: time to first token, key-field extraction accuracy, and total processing time.

AVX-512CPU inferenceDeepSeek-V4-Flash
0 likes · 12 min read
DeepSeek-V4-Flash on 64GB CPU: Why 96GB Xeon Can't Load It
Geek Labs
Geek Labs
Jul 14, 2026 · Artificial Intelligence

How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM

Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.

CPU inferenceGLM-5.2INT4 quantization
0 likes · 7 min read
How Colibri Runs a 744 B MoE Model on CPU with Only 25 GB RAM
Black & White Path
Black & White Path
Apr 8, 2026 · Artificial Intelligence

Run Massive AI Models on a Single PC: The 1‑Bit LLM Revolution

Microsoft’s open‑source bitnet.cpp transforms 100‑billion‑parameter LLM inference from GPU‑only to ordinary CPUs by replacing floating‑point matrix multiplication with integer add‑subtract, cutting energy use by 82 %, memory by 90 % and delivering up to 6× speed on x86/ARM hardware.

1-bit LLMBitNetCPU inference
0 likes · 7 min read
Run Massive AI Models on a Single PC: The 1‑Bit LLM Revolution
AI Explorer
AI Explorer
Mar 17, 2026 · Artificial Intelligence

Microsoft Open‑Sources BitNet: 1‑Bit Inference Framework Runs Billion‑Parameter Models on CPUs with Up to 6× Speedup

BitNet.cpp, Microsoft’s open‑source 1‑bit inference engine, enables billion‑parameter language models to run on ordinary CPUs, delivering 1.37‑6.17× speed improvements and 55‑82% energy reductions across ARM and x86 platforms, while providing a simple three‑step build‑and‑run workflow and broad hardware support.

1-bit quantizationBitNetCPU inference
0 likes · 8 min read
Microsoft Open‑Sources BitNet: 1‑Bit Inference Framework Runs Billion‑Parameter Models on CPUs with Up to 6× Speedup
AIWalker
AIWalker
Feb 27, 2026 · Artificial Intelligence

YOLO26 Review: End-to-End, NMS‑Free Edge AI Boosts CPU Inference by 43%

This article analyzes YOLO26’s architecture redesign that eliminates NMS, removes DFL, introduces progressive loss balancing, STAL, and the MuSGD optimizer, achieving up to 43% faster CPU inference and simplifying deployment for edge vision tasks across detection, segmentation, classification, pose estimation, and OBB.

CPU inferenceNMS-freeYOLO26
0 likes · 13 min read
YOLO26 Review: End-to-End, NMS‑Free Edge AI Boosts CPU Inference by 43%
Weekly Large Model Application
Weekly Large Model Application
Feb 22, 2026 · Artificial Intelligence

2026 Guide: Pure‑CPU Open‑Source Chinese TTS Models Optimized for Performance

This article reviews the most capable open‑source Chinese text‑to‑speech models that run entirely on CPU in 2026, compares their quantization and speed features, recommends acceleration engines, outlines five hard‑won optimization rules, and provides a concise selection guide for various deployment scenarios.

CPU inferenceChinese TTSONNX Runtime
0 likes · 6 min read
2026 Guide: Pure‑CPU Open‑Source Chinese TTS Models Optimized for Performance
Weekly Large Model Application
Weekly Large Model Application
Feb 22, 2026 · Artificial Intelligence

2026 Guide to Running Open‑Source ASR on Pure CPU

The 2026 overview details lightweight, heavily quantized open‑source speech‑recognition models and CPU‑specific inference engines, offering concrete tips, model comparisons, and a concise selection guide that enable real‑time, GPU‑free ASR deployment with low latency and high stability.

ASRCPU inferenceSpeech Recognition
0 likes · 4 min read
2026 Guide to Running Open‑Source ASR on Pure CPU
Old Meng AI Explorer
Old Meng AI Explorer
Dec 29, 2025 · Artificial Intelligence

Run 100B LLMs on a Laptop: How BitNet’s 1‑bit Quantization Makes It Possible

BitNet’s 1‑bit quantization shrinks model size and compute needs by tenfold, enabling ordinary CPUs and low‑power ARM devices to run 2B‑100B language models locally with acceptable speed, low power consumption, and near‑original quality, while providing simple installation and optional GPU acceleration.

BitNetCPU inferenceLLM quantization
0 likes · 10 min read
Run 100B LLMs on a Laptop: How BitNet’s 1‑bit Quantization Makes It Possible
Old Meng AI Explorer
Old Meng AI Explorer
Dec 25, 2025 · Artificial Intelligence

Run 100B LLM on a Laptop: BitNet’s 1‑Bit Quantization Enables CPU‑Only AI

BitNet, Microsoft’s open‑source 1‑bit quantization framework, shrinks model size by up to ten‑fold and lets ordinary CPUs—including i7 laptops and ARM tablets—run 2B‑100B language models at usable speeds while cutting power consumption dramatically, offering a practical, GPU‑free solution for local AI.

BitNetCPU inferenceLLM quantization
0 likes · 9 min read
Run 100B LLM on a Laptop: BitNet’s 1‑Bit Quantization Enables CPU‑Only AI
DataFunTalk
DataFunTalk
Apr 19, 2025 · Artificial Intelligence

Microsoft Research's Open‑Source Native 1‑Bit LLM BitNet b1.58 2B4T: Design, Performance, and Deployment

Microsoft Research released BitNet b1.58 2B4T, the first open‑source native 1‑bit large language model with 2 billion parameters, 1.58‑bit effective precision and a 0.4 GB footprint, achieving full‑precision performance while enabling efficient CPU and GPU inference for edge AI applications.

1-bit quantizationCPU inferenceLLM
0 likes · 10 min read
Microsoft Research's Open‑Source Native 1‑Bit LLM BitNet b1.58 2B4T: Design, Performance, and Deployment