Tagged articles
inference engine
26 articles · Page 1 of 1
This article details how dynamic parameters function in OntoL-Manage: they are entity-level runtime JSON parameters stored in action_param, distinct from DSL replacement parameters and documentation-only reference fields, enabling a read-compute-write loop via behavior function chains.
DSLOntoL-Managebehavior ontology
0 likes · 4 min read

Strata inference engine enables local deployment of DeepSeek-V4.1-Flash on consumer GPUs by tiering model weights across VRAM, system RAM, and NVMe storage, achieving 9–11 tokens/s on an RTX 3060 12GB with 46GB RAM using 2-bit quantization and speculative decoding.
DeepSeek V4.1 FlashMoEStrata
0 likes · 9 min read

This article details how OntoL's custom DSL and inference engine collaborate with standard fields and sandbox replicas to transform static ontologies into executable knowledge graphs, covering architecture layers, rule structure, execution flow, propagation, LLM integration, and engineering challenges.
DSLKnowledge GraphOntoL
0 likes · 35 min read

DeepSeek has open-sourced its production inference engine DeepInfer and token codec component deepseek-recipe, revealing a single-cluster throughput of 155 trillion tokens per day and $38 per GPU per hour revenue, signaling further reductions in per-token inference pricing for developers.
AI infrastructureDeepInferDeepSeek
0 likes · 3 min read

Inco AI's open-source inference engine Splash 1.4.0 achieves 120 tokens/second decode and 1.3K tokens/second prefill on a 64GB M5 Pro running Qwen3.8-Flash-Next, outperforming Ollama by 3x and Strata by 30%, with MoE activation efficiency enabling large models on modest memory.
Apple SiliconBenchmarkInco AI
0 likes · 7 min read

This article details SGLang's device plugin mechanism (Platform and Hook) enabling hardware-agnostic inference, the SGLang-Kunlun plugin's cuda-like compatibility layer, a five-step standardized adaptation process with measurable acceptance criteria, and AI-driven Harness automation that reduced DeepSeek V4 Flash version upgrades to two hours while achieving 1.55x operator speedups via dual-stream vector/matrix compute overlap.
AI-assisted DevelopmentCUDA GraphDevice Plugin
0 likes · 22 min read

This article reviews three cutting-edge AI open-source projects: DeepSeek's plugin-based Agent runtime framework (deepseek-harness), archify for generating verifiable architecture diagrams directly in coding agents, and colibri, a pure C inference engine enabling 744B MoE models to run on 25GB RAM via disk streaming.
DeepSeekMoEPlugin Architecture
0 likes · 8 min read

Colibri, a pure C inference engine with zero dependencies, enables running the 744B parameter GLM-5.2 MoE model on consumer hardware with just 25GB RAM and NVMe SSD by leveraging MoE sparsity and a three-tier storage scheduling system across VRAM, RAM, and disk, achieving 0.05–6.8 tok/s depending on hardware.
ColibriGLM-5.2MoE
0 likes · 14 min read

A pure C inference engine runs the 2.78 trillion parameter Kimi K3 model on just 8GB RAM by streaming weights from disk, leveraging MoE sparsity (only 3.7% active per token) and computing directly on compressed formats, achieving correct output at 32.69 seconds per token on consumer hardware.
C languageKimi K3LLM compression
0 likes · 11 min read

OntoL uses dual ontology modeling — separating business material flow from financial cost recognition — to decouple warehouse issuance from profit-and-loss impact, enabling automatic WIP tracking, real-time discrepancy detection, and full audit traceability, cutting month-end closing from seven days to half a day in manufacturing case studies.
TBox/ABoxWIP cost accountingaudit traceability
0 likes · 13 min read

This article walks through the core concepts needed before deploying a large language model on‑premises, covering weight precision, quantization methods, model packaging formats, inference engines, GPU memory considerations, KV‑cache sizing, sampling strategies, optional extensions such as LoRA and RAG, and a step‑by‑step decision workflow to match hardware, model, and deployment goals.
KV cacheLLMdeployment
0 likes · 21 min read

The article presents a comprehensive, production‑ready guide for the L1 layer of AI infrastructure, detailing how model gateways unify calls, intelligent routing selects the optimal model, inference engines maximize GPU throughput, and quantization and KV‑Cache techniques dramatically cut costs while maintaining performance.
AI infrastructureKV cacheLLM routing
0 likes · 25 min read

The vLLM 0.24.0 update adds MiniMax‑M3, DeepSeek‑V4, DiffusionGemma support, a Streaming Parser Engine, and a new device_ids parameter, delivering faster inference, lower memory use, and broader hardware compatibility for large‑model deployments.
DiffusionGemmaMiniMax M3PagedAttention
0 likes · 9 min read

The article presents a detailed architecture for a high‑reliability cognitive reasoning system that combines logical inference, semantic constraints, and a seven‑layer defense to achieve efficient deduction and strict error prevention across critical domains such as medical diagnosis and financial risk control.
Knowledge GraphOntologycognitive reasoning
0 likes · 6 min read

Redis founder antirez unveiled ds4.c, a pure‑C inference engine that leverages Objective‑C and Metal to run DeepSeek V4 locally on Mac devices, delivering about 27 token/s on an M3 Ultra—far slower than GPU servers but offering a dependency‑free, on‑device solution that keeps data private.
AIC++DeepSeek
0 likes · 8 min read

This article walks through constructing Mini-vLLM, a from‑scratch LLM inference engine that tackles the O(N²) attention cost with KV‑cache, boosts throughput via dynamic batching, adds observability with Prometheus/Grafana, supports gRPC, and scales across multiple workers, with benchmark numbers demonstrating its CPU‑only performance.
DockerKV cacheLLM
0 likes · 12 min read


JavaEdgeJun 27, 2025 · Artificial Intelligence
The article explains what inference engines are, why they are needed beyond raw Python scripts, and outlines best practices such as model quantization, batching, and parallelism, while comparing popular open‑source and commercial options for production AI workloads.
AI DeploymentLLMParallelism
0 likes · 14 min read

This article examines the importance of domestic large‑model inference engines, compares Chinese and international chips, evaluates four architectural approaches, discusses practical challenges such as performance loss and model support, and outlines future expectations for high‑performance, heterogeneous‑chip inference solutions.
domestic chipinference enginelarge model
0 likes · 9 min read

The talk presents an overview of the MNN inference engine, outlines the end‑to‑end workflow for deploying large language models on mobile devices, discusses technical challenges and practical solutions, and concludes with future directions for edge AI deployment.
AIEdge DeploymentMNN
0 likes · 2 min read

ShaderNN is an open‑source, sub‑2 MB GPU‑shader inference engine that runs TensorFlow, PyTorch and ONNX models directly on mobile graphics textures via OpenGL fragment and compute shaders, delivering real‑time, low‑power AI for image‑heavy tasks while eliminating third‑party dependencies and achieving up to 90 % speed gains.
GPUShaderinference engine
0 likes · 11 min read

Pitaya, built by ByteDance’s Client AI and MLX teams, is a comprehensive end‑side AI engineering platform that provides a full workflow from model development and data preparation to deployment, monitoring, and federated learning, supporting large‑scale commercial scenarios across multiple apps.
AI PlatformFederated Learningedge AI
0 likes · 14 min read

The article presents a comprehensive overview of Baidu's open‑source deep learning platform PaddlePaddle, detailing its full‑stack architecture, core technologies such as unified dynamic‑static graph, large‑scale distributed training, multi‑platform inference, an extensive model zoo, hardware adaptation, and showcases a real‑world deployment case in power‑grid monitoring.
AI frameworkPaddlePaddledistributed training
0 likes · 15 min read

Alibaba’s MNN, a lightweight high‑performance deep‑learning inference engine, earned top honors in China’s 2022 “Science & Innovation China” awards, and delivers impressive gains such as 350% speedup on X86 CPUs, 2.1‑2.3× acceleration on ARM with sparse models, plus integrated OpenCV/Numpy functionality for edge AI deployment.
AI DeploymentAlibabaMNN
0 likes · 4 min read

This article explains on‑device machine learning concepts, compares front‑end inference engines such as TensorFlow.js, ONNX.js and WebDNN across CPU, WASM and WebGL, and presents practical optimization techniques like vectorization, memory layout, graph fusion and mixed‑precision to boost performance for real‑time applications.
frontendinference enginemachine learning
0 likes · 11 min read

Alibaba’s MNN (Mobile Neural Network) engine, now open‑sourced on GitHub, showcases how a lightweight, end‑side deep‑learning inference framework tackles fragmentation, optimizes model conversion, scheduling, and execution across diverse devices, delivering significant performance gains for mobile and IoT AI applications.
MNNOperator Fusiongraph optimization
0 likes · 15 min read

Alibaba’s massive product knowledge graph combines billions of triples, AI‑driven inference, and semantic reasoning to enable millisecond‑level, explainable detection of illegal or counterfeit items across its e‑commerce ecosystem, improving platform governance and consumer experience.
AIAlibabaKnowledge Graph
0 likes · 8 min read
