Skip to content
Tagged articles

inference engine

26 articles · Page 1 of 1
AI Large-Model Wave and Transformation Guide
AI Large-Model Wave and Transformation Guide
Oct 28, 2026 · Backend Development

OntoL's DSL and Inference Engine: From Static Ontology to Executable Knowledge Graphs

This article details how OntoL's custom DSL and inference engine collaborate with standard fields and sandbox replicas to transform static ontologies into executable knowledge graphs, covering architecture layers, rule structure, execution flow, propagation, LLM integration, and engineering challenges.

DSLKnowledge GraphOntoL
0 likes · 35 min read
OntoL's DSL and Inference Engine: From Static Ontology to Executable Knowledge Graphs
JavaScript
JavaScript
Oct 21, 2026 · Artificial Intelligence

DeepSeek Open-Sources DeepInfer Inference Engine: 155T Tokens/Day, $38/GPU/Hour

DeepSeek has open-sourced its production inference engine DeepInfer and token codec component deepseek-recipe, revealing a single-cluster throughput of 155 trillion tokens per day and $38 per GPU per hour revenue, signaling further reductions in per-token inference pricing for developers.

AI infrastructureDeepInferDeepSeek
0 likes · 3 min read
DeepSeek Open-Sources DeepInfer Inference Engine: 155T Tokens/Day, $38/GPU/Hour
Lao Guo's Learning Space
Lao Guo's Learning Space
Oct 21, 2026 · Artificial Intelligence

Splash 1.4.0: 120 tok/s on 64GB M5 Pro, 3x Faster Than Ollama

Inco AI's open-source inference engine Splash 1.4.0 achieves 120 tokens/second decode and 1.3K tokens/second prefill on a 64GB M5 Pro running Qwen3.8-Flash-Next, outperforming Ollama by 3x and Strata by 30%, with MoE activation efficiency enabling large models on modest memory.

Apple SiliconBenchmarkInco AI
0 likes · 7 min read
Splash 1.4.0: 120 tok/s on 64GB M5 Pro, 3x Faster Than Ollama
Baidu Intelligent Cloud Tech Hub
Baidu Intelligent Cloud Tech Hub
Sep 29, 2026 · Backend Development

SGLang Multi-Hardware Plugin Architecture and Kunlun Chip Adaptation: A 2-Hour Model Upgrade Case Study

This article details SGLang's device plugin mechanism (Platform and Hook) enabling hardware-agnostic inference, the SGLang-Kunlun plugin's cuda-like compatibility layer, a five-step standardized adaptation process with measurable acceptance criteria, and AI-driven Harness automation that reduced DeepSeek V4 Flash version upgrades to two hours while achieving 1.55x operator speedups via dual-stream vector/matrix compute overlap.

AI-assisted DevelopmentCUDA GraphDevice Plugin
0 likes · 22 min read
SGLang Multi-Hardware Plugin Architecture and Kunlun Chip Adaptation: A 2-Hour Model Upgrade Case Study
Architecture Digest
Architecture Digest
Sep 21, 2026 · Artificial Intelligence

3 AI Open-Source Projects: DeepSeek Agent Runtime, Verified Diagrams, 744B MoE on 25GB RAM

This article reviews three cutting-edge AI open-source projects: DeepSeek's plugin-based Agent runtime framework (deepseek-harness), archify for generating verifiable architecture diagrams directly in coding agents, and colibri, a pure C inference engine enabling 744B MoE models to run on 25GB RAM via disk streaming.

DeepSeekMoEPlugin Architecture
0 likes · 8 min read
3 AI Open-Source Projects: DeepSeek Agent Runtime, Verified Diagrams, 744B MoE on 25GB RAM
AI Architecture Path
AI Architecture Path
Sep 17, 2026 · Artificial Intelligence

Run 744B MoE Model on 25GB RAM: Colibri's Tiered Storage Breakthrough

Colibri, a pure C inference engine with zero dependencies, enables running the 744B parameter GLM-5.2 MoE model on consumer hardware with just 25GB RAM and NVMe SSD by leveraging MoE sparsity and a three-tier storage scheduling system across VRAM, RAM, and disk, achieving 0.05–6.8 tok/s depending on hardware.

ColibriGLM-5.2MoE
0 likes · 14 min read
Run 744B MoE Model on 25GB RAM: Colibri's Tiered Storage Breakthrough
Architecture Digest
Architecture Digest
Sep 13, 2026 · Artificial Intelligence

Running 2.78T-Parameter Kimi K3 on 8GB RAM: Pure C Inference Engine Explained

A pure C inference engine runs the 2.78 trillion parameter Kimi K3 model on just 8GB RAM by streaming weights from disk, leveraging MoE sparsity (only 3.7% active per token) and computing directly on compressed formats, achieving correct output at 32.69 seconds per token on consumer hardware.

C languageKimi K3LLM compression
0 likes · 11 min read
Running 2.78T-Parameter Kimi K3 on 8GB RAM: Pure C Inference Engine Explained
AI Large-Model Wave and Transformation Guide
AI Large-Model Wave and Transformation Guide
Sep 11, 2026 · Industry Insights

Why Material Issuance ≠ Cost Transfer: OntoL's Ontology Modeling Fixes Business-Finance Disconnect

OntoL uses dual ontology modeling — separating business material flow from financial cost recognition — to decouple warehouse issuance from profit-and-loss impact, enabling automatic WIP tracking, real-time discrepancy detection, and full audit traceability, cutting month-end closing from seven days to half a day in manufacturing case studies.

TBox/ABoxWIP cost accountingaudit traceability
0 likes · 13 min read
Why Material Issuance ≠ Cost Transfer: OntoL's Ontology Modeling Fixes Business-Finance Disconnect
AI Engineer Programming
AI Engineer Programming
Aug 21, 2026 · Artificial Intelligence

Essential Concepts and Terminology for Deploying Large Language Models Locally

This article walks through the core concepts needed before deploying a large language model on‑premises, covering weight precision, quantization methods, model packaging formats, inference engines, GPU memory considerations, KV‑cache sizing, sampling strategies, optional extensions such as LoRA and RAG, and a step‑by‑step decision workflow to match hardware, model, and deployment goals.

KV cacheLLMdeployment
0 likes · 21 min read
Essential Concepts and Terminology for Deploying Large Language Models Locally
ThinkingAgent
ThinkingAgent
Jul 5, 2026 · Artificial Intelligence

Building L1 AI Infra: Model Gateways, Smart Routing, and High‑Performance Inference Engines

The article presents a comprehensive, production‑ready guide for the L1 layer of AI infrastructure, detailing how model gateways unify calls, intelligent routing selects the optimal model, inference engines maximize GPU throughput, and quantization and KV‑Cache techniques dramatically cut costs while maintaining performance.

AI infrastructureKV cacheLLM routing
0 likes · 25 min read
Building L1 AI Infra: Model Gateways, Smart Routing, and High‑Performance Inference Engines
AI Large-Model Wave and Transformation Guide
AI Large-Model Wave and Transformation Guide
Jun 8, 2026 · Artificial Intelligence

Designing a High‑Reliability Cognitive Reasoning System with Ontology‑Based Architecture

The article presents a detailed architecture for a high‑reliability cognitive reasoning system that combines logical inference, semantic constraints, and a seven‑layer defense to achieve efficient deduction and strict error prevention across critical domains such as medical diagnosis and financial risk control.

Knowledge GraphOntologycognitive reasoning
0 likes · 6 min read
Designing a High‑Reliability Cognitive Reasoning System with Ontology‑Based Architecture
DeepHub IMBA
DeepHub IMBA
Apr 4, 2026 · Artificial Intelligence

Building Mini-vLLM from Scratch: KV‑Cache, Dynamic Batching, and Distributed Inference

This article walks through constructing Mini-vLLM, a from‑scratch LLM inference engine that tackles the O(N²) attention cost with KV‑cache, boosts throughput via dynamic batching, adds observability with Prometheus/Grafana, supports gRPC, and scales across multiple workers, with benchmark numbers demonstrating its CPU‑only performance.

DockerKV cacheLLM
0 likes · 12 min read
Building Mini-vLLM from Scratch: KV‑Cache, Dynamic Batching, and Distributed Inference
DataFunSummit
DataFunSummit
Dec 24, 2024 · Artificial Intelligence

Considerations and Practices for Domesticating Large‑Model Inference Engines

This article examines the importance of domestic large‑model inference engines, compares Chinese and international chips, evaluates four architectural approaches, discusses practical challenges such as performance loss and model support, and outlines future expectations for high‑performance, heterogeneous‑chip inference solutions.

domestic chipinference enginelarge model
0 likes · 9 min read
Considerations and Practices for Domesticating Large‑Model Inference Engines
DataFunSummit
DataFunSummit
Sep 11, 2023 · Artificial Intelligence

Challenges and Insights for Deploying Large Models on Edge with MNN

The talk presents an overview of the MNN inference engine, outlines the end‑to‑end workflow for deploying large language models on mobile devices, discusses technical challenges and practical solutions, and concludes with future directions for edge AI deployment.

AIEdge DeploymentMNN
0 likes · 2 min read
Challenges and Insights for Deploying Large Models on Edge with MNN
OPPO Kernel Craftsman
OPPO Kernel Craftsman
Oct 28, 2022 · Artificial Intelligence

ShaderNN: A GPU Shader‑Based Lightweight Inference Engine for Mobile AI Applications

ShaderNN is an open‑source, sub‑2 MB GPU‑shader inference engine that runs TensorFlow, PyTorch and ONNX models directly on mobile graphics textures via OpenGL fragment and compute shaders, delivering real‑time, low‑power AI for image‑heavy tasks while eliminating third‑party dependencies and achieving up to 90 % speed gains.

GPUShaderinference engine
0 likes · 11 min read
ShaderNN: A GPU Shader‑Based Lightweight Inference Engine for Mobile AI Applications
ByteDance Terminal Technology
ByteDance Terminal Technology
Jul 29, 2022 · Artificial Intelligence

Pitaya: ByteDance’s End‑Side AI Engineering Platform Overview

Pitaya, built by ByteDance’s Client AI and MLX teams, is a comprehensive end‑side AI engineering platform that provides a full workflow from model development and data preparation to deployment, monitoring, and federated learning, supporting large‑scale commercial scenarios across multiple apps.

AI PlatformFederated Learningedge AI
0 likes · 14 min read
Pitaya: ByteDance’s End‑Side AI Engineering Platform Overview
DataFunTalk
DataFunTalk
Apr 14, 2022 · Artificial Intelligence

PaddlePaddle Deep Learning Platform: Architecture, Core Technologies, and Real‑World Applications

The article presents a comprehensive overview of Baidu's open‑source deep learning platform PaddlePaddle, detailing its full‑stack architecture, core technologies such as unified dynamic‑static graph, large‑scale distributed training, multi‑platform inference, an extensive model zoo, hardware adaptation, and showcases a real‑world deployment case in power‑grid monitoring.

AI frameworkPaddlePaddledistributed training
0 likes · 15 min read
PaddlePaddle Deep Learning Platform: Architecture, Core Technologies, and Real‑World Applications
DaTaobao Tech
DaTaobao Tech
Mar 11, 2022 · Artificial Intelligence

How Alibaba’s MNN Engine Achieves 350% CPU Speedup and Sparse Acceleration

Alibaba’s MNN, a lightweight high‑performance deep‑learning inference engine, earned top honors in China’s 2022 “Science & Innovation China” awards, and delivers impressive gains such as 350% speedup on X86 CPUs, 2.1‑2.3× acceleration on ARM with sparse models, plus integrated OpenCV/Numpy functionality for edge AI deployment.

AI DeploymentAlibabaMNN
0 likes · 4 min read
How Alibaba’s MNN Engine Achieves 350% CPU Speedup and Sparse Acceleration
Alibaba Terminal Technology
Alibaba Terminal Technology
Feb 3, 2021 · Frontend Development

How Front-End AI Inference Engines Achieve Real-Time Smart Recognition

This article explains on‑device machine learning concepts, compares front‑end inference engines such as TensorFlow.js, ONNX.js and WebDNN across CPU, WASM and WebGL, and presents practical optimization techniques like vectorization, memory layout, graph fusion and mixed‑precision to boost performance for real‑time applications.

frontendinference enginemachine learning
0 likes · 11 min read
How Front-End AI Inference Engines Achieve Real-Time Smart Recognition
Alibaba Cloud Developer
Alibaba Cloud Developer
Jul 2, 2019 · Artificial Intelligence

How MNN Powers Mobile AI: Inside Alibaba’s Open‑Source Inference Engine

Alibaba’s MNN (Mobile Neural Network) engine, now open‑sourced on GitHub, showcases how a lightweight, end‑side deep‑learning inference framework tackles fragmentation, optimizes model conversion, scheduling, and execution across diverse devices, delivering significant performance gains for mobile and IoT AI applications.

MNNOperator Fusiongraph optimization
0 likes · 15 min read
How MNN Powers Mobile AI: Inside Alibaba’s Open‑Source Inference Engine