Tagged articles
CPU inference
15 articles · Page 1 of 1
The llama-modchip project demonstrates running a 284B parameter MoE model on a 64GB RAM CPU-only machine via aggressive quantization to 56.6GB and kernel optimizations, but achieves only 9 tokens/sec decoding speed, revealing that memory capacity—not GPU—is becoming the key bottleneck for local large language model inference, with practical implications for Mac memory sizing.
CPU inferenceMac hardwareMoE
0 likes · 11 min read

The llama-modchip project runs a 284B-parameter DeepSeek-V4-Flash model on a 64GB RAM CPU-only machine via aggressive quantization to 1.71 bits/weight (56.56 GiB), achieving 9–10 tokens/sec, but a 96GB Xeon server fails due to custom GGUF tensor types and missing AVX-512, highlighting hardware-software compatibility gaps.
AVX-512CPU inferenceDeepSeek-V4-Flash
0 likes · 13 min read

The article explains how llama-modchip runs 284B DeepSeek-V4-Flash on a 64GB RAM CPU-only machine via aggressive quantization (1.71 bits/weight), while a 96GB Xeon server fails due to tensor format mismatch and missing AVX-512. It breaks down inference latency for long contracts (~10 minutes for 20k tokens) and proposes three evaluation metrics: time to first token, key-field extraction accuracy, and total processing time.
AVX-512CPU inferenceDeepSeek-V4-Flash
0 likes · 12 min read

colibri, a zero-dependency C inference engine, runs Zhipu's 744B-parameter GLM-5.2 MoE model on a 25GB RAM laptop by keeping dense weights in memory and streaming routed experts from NVMe, achieving ~1 token/s after cache warm-up with router-lookahead prefetching and learned caching.
AVX2CPU inferenceColibri
0 likes · 11 min read


Geek LabsJul 14, 2026 · Artificial Intelligence
Colibri demonstrates that the 744‑billion‑parameter GLM‑5.2 MoE model can be executed on a pure‑CPU system using roughly 25 GB of memory by streaming expert weights from disk, int4‑quantizing dense parameters, and employing a compact C inference engine.
CPU inferenceGLM-5.2INT4 quantization
0 likes · 7 min read

Microsoft’s open‑source bitnet.cpp transforms 100‑billion‑parameter LLM inference from GPU‑only to ordinary CPUs by replacing floating‑point matrix multiplication with integer add‑subtract, cutting energy use by 82 %, memory by 90 % and delivering up to 6× speed on x86/ARM hardware.
1-bit LLMBitNetCPU inference
0 likes · 7 min read

BitNet.cpp, Microsoft’s open‑source 1‑bit inference engine, enables billion‑parameter language models to run on ordinary CPUs, delivering 1.37‑6.17× speed improvements and 55‑82% energy reductions across ARM and x86 platforms, while providing a simple three‑step build‑and‑run workflow and broad hardware support.
1-bit quantizationBitNetCPU inference
0 likes · 8 min read


AIWalkerFeb 27, 2026 · Artificial Intelligence
This article analyzes YOLO26’s architecture redesign that eliminates NMS, removes DFL, introduces progressive loss balancing, STAL, and the MuSGD optimizer, achieving up to 43% faster CPU inference and simplifying deployment for edge vision tasks across detection, segmentation, classification, pose estimation, and OBB.
CPU inferenceNMS-freeYOLO26
0 likes · 13 min read

This article explains how to build a cost‑effective, CPU‑only face verification system for small‑scale projects using the lightweight YuNet detector and SFace recognizer, covering the models’ principles, implementation steps with OpenCV and Gradio, and performance considerations.
CPU inferenceFace DetectionGradio
0 likes · 7 min read

This article reviews the most capable open‑source Chinese text‑to‑speech models that run entirely on CPU in 2026, compares their quantization and speed features, recommends acceleration engines, outlines five hard‑won optimization rules, and provides a concise selection guide for various deployment scenarios.
CPU inferenceChinese TTSONNX Runtime
0 likes · 6 min read

The 2026 overview details lightweight, heavily quantized open‑source speech‑recognition models and CPU‑specific inference engines, offering concrete tips, model comparisons, and a concise selection guide that enable real‑time, GPU‑free ASR deployment with low latency and high stability.
ASRCPU inferenceSpeech Recognition
0 likes · 4 min read

BitNet’s 1‑bit quantization shrinks model size and compute needs by tenfold, enabling ordinary CPUs and low‑power ARM devices to run 2B‑100B language models locally with acceptable speed, low power consumption, and near‑original quality, while providing simple installation and optional GPU acceleration.
BitNetCPU inferenceLLM quantization
0 likes · 10 min read

BitNet, Microsoft’s open‑source 1‑bit quantization framework, shrinks model size by up to ten‑fold and lets ordinary CPUs—including i7 laptops and ARM tablets—run 2B‑100B language models at usable speeds while cutting power consumption dramatically, offering a practical, GPU‑free solution for local AI.
BitNetCPU inferenceLLM quantization
0 likes · 9 min read

Microsoft Research released BitNet b1.58 2B4T, the first open‑source native 1‑bit large language model with 2 billion parameters, 1.58‑bit effective precision and a 0.4 GB footprint, achieving full‑precision performance while enabling efficient CPU and GPU inference for edge AI applications.
1-bit quantizationCPU inferenceLLM
0 likes · 10 min read

This guide walks you through deploying the DeepSeek‑R1‑Distill model on Volcengine CPU ECS instances, covering use‑case scenarios, recommended server types, Docker setup, environment configuration, and verification steps to achieve cost‑effective, high‑compatibility AI inference.
AI model deploymentCPU inferenceDeepSeek
0 likes · 6 min read
