Skip to content
Tagged articles

Inference Optimization

114 articles · Page 1 of 2
SmartApe AI
SmartApe AI
Nov 15, 2026 · Artificial Intelligence

Tencent Hy4 Flash: 12B Active Params, 1/8 DeepSeek Cost, Matches Claude Haiku 5.5

Tencent's Hy4 Flash, a 224B MoE model with 12B active parameters, achieves 150+ tokens/s decoding speed at 1/20 the cost of its predecessor, outperforms DeepSeek-V4.1-Flash on cost efficiency, and matches Claude Haiku 5.5 user satisfaction in blind tests across 259 office tasks.

HunyuanHy4 FlashInference Optimization
0 likes · 6 min read
Tencent Hy4 Flash: 12B Active Params, 1/8 DeepSeek Cost, Matches Claude Haiku 5.5
SmartApe AI
SmartApe AI
Nov 6, 2026 · Artificial Intelligence

Qwen Sparse Attention Ported to Qwen3.8-27B: 2.9x Prefill, 1.8x Decode Speedup

A Japanese developer implemented Qwen Sparse Attention (QSA) on Qwen3.8-27B, achieving 2.92x faster prefill and 1.79x faster decode at 261K tokens while maintaining perplexity and DocVQA scores; a quantized version further reaches 2375 tok/s prefill and 115 tok/s decode with 22.8 GiB VRAM.

Community DevelopmentInference OptimizationQSA
0 likes · 5 min read
Qwen Sparse Attention Ported to Qwen3.8-27B: 2.9x Prefill, 1.8x Decode Speedup
Su San Talks Tech
Su San Talks Tech
Oct 30, 2026 · Artificial Intelligence

DeepSeek's 10 Game-Changing Open-Source Projects: From GRPO to Agent Infrastructure

This article reviews DeepSeek's 10 most impactful open-source projects, spanning math reasoning (GRPO), MoE communication (DeepEP), distributed storage (3FS), Hopper-optimized kernels (FlashMLA), production inference engine (DeepInfer), optical document compression (DeepSeek-OCR), code generation (Coder-V2), pure RL reasoning (R1), efficient Mixture-of-Experts training (V3), and next-gen agent infrastructure (Harness).

DeepSeekGRPOInference Optimization
0 likes · 25 min read
DeepSeek's 10 Game-Changing Open-Source Projects: From GRPO to Agent Infrastructure
SmartApe AI
SmartApe AI
Oct 28, 2026 · Artificial Intelligence

Qwen 4 Imminent: Leaked Code & Anonymous Models Reveal Architecture Details

Community clues suggest Alibaba's Qwen 4 is near release, with inference optimizations in vLLM and llama.cpp, architectural details like a 27B dense model with 5120 hidden dim, 24 query heads, 4 KV heads, 4 residual streams, MTP, and QSA sparse attention, while anonymous Arena models Kalmia and Kerria are speculated as Qwen 4 variants showing strong code and 3D generation.

AlibabaArena benchmarkFP8 quantization
0 likes · 7 min read
Qwen 4 Imminent: Leaked Code & Anonymous Models Reveal Architecture Details
SmartApe AI
SmartApe AI
Oct 23, 2026 · Artificial Intelligence

Agent Cache Engineering: Tsinghua & CUHK's AKV Cuts Inference Cost 27.7% via RSI

Tsinghua and CUHK researchers introduce Cache Engineering, a paradigm where AI agents directly manage KV cache via Append, Drop, and Repos operations, avoiding re-prefill costs; their open-source AKV system achieves 27.7% inference cost reduction and 1.7% higher task scores, while recursive self-improvement with MiniMax-M2.7 discovers even better cache policies.

AI AgentsAKV SystemCUHK
0 likes · 12 min read
Agent Cache Engineering: Tsinghua & CUHK's AKV Cuts Inference Cost 27.7% via RSI
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Oct 15, 2026 · Artificial Intelligence

Day-0 Adaptation of Step 5 Preview on Zhenwu M890 Supernode: 3 Key Optimizations

Alibaba Cloud's Zhenwu M890 supernode (64 chips, 800GB/s interconnect, 9TB unified memory) adapted StepFun's Step 5 Preview (600B sparse MoE, 27B active, 1M context, multimodal) with operator fusion, expert parallel optimization, and prefill-decode separation, achieving lower TTFT, higher decode throughput, and stable 1M context support.

Inference OptimizationMoEStep 5 Preview
0 likes · 6 min read
Day-0 Adaptation of Step 5 Preview on Zhenwu M890 Supernode: 3 Key Optimizations
JD Retail Technology
JD Retail Technology
Oct 15, 2026 · Artificial Intelligence

RA-CFGCache: Risk-Aligned Caching for CFG-Guided Diffusion Models

RA-CFGCache introduces a risk-aligned caching framework for classifier-free guided diffusion models that jointly models cross-branch error alignment and timestep-dependent error propagation, achieving superior efficiency-fidelity trade-offs on FLUX.1-dev, Wan2.1, and CogVideoX while remaining compatible with existing cache estimators.

CachingClassifier-Free GuidanceGenerative AI
0 likes · 19 min read
RA-CFGCache: Risk-Aligned Caching for CFG-Guided Diffusion Models
Architects' Tech Alliance
Architects' Tech Alliance
Oct 14, 2026 · Artificial Intelligence

AI Super Nodes: China's 2026 Compute Framework & NVIDIA Vera Rubin Deep Dive

This analysis examines AI super node architecture as the solution to large model training and inference bottlenecks, detailing NVIDIA's Vera Rubin NVL72 platform with 7 chips and 5 rack types, China's domestic super node solutions from Huawei, Sugon, Alibaba, and Baidu, and projects a 341% CAGR reaching 1.1 trillion RMB by 2028.

AI super-nodeInference OptimizationNVIDIA Vera Rubin
0 likes · 10 min read
AI Super Nodes: China's 2026 Compute Framework & NVIDIA Vera Rubin Deep Dive
DataFunTalk
DataFunTalk
Oct 2, 2026 · Artificial Intelligence

Xiaohongshu's GR-Inference: Custom Engine for 3.6x Faster Generative Retrieval

Xiaohongshu built a custom inference engine GR-Inference for generative search retrieval, addressing unique load characteristics — long context, short decode, large dynamic beam, and constrained generation — that break general frameworks, achieving 1.5–3.6x throughput over SGLang and improving recall and click-through rates.

GR-InferenceInference OptimizationLLM serving
0 likes · 7 min read
Xiaohongshu's GR-Inference: Custom Engine for 3.6x Faster Generative Retrieval
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Sep 29, 2026 · Artificial Intelligence

AI That Builds AI: Naive AI's Self-Evolving Models Beat Top Benchmarks in 151 Experiments

Naive AI demonstrates AI-driven R&D where agents autonomously redesign model architectures, optimize training systems, and achieve breakthrough benchmarks—Naive-N0.5-Flash surpasses larger models on coding and research tasks, while AI-built NaiveRT delivers 30-50x inference speedups and AutoWM sets new world model records through self-discovered algorithms.

AI for AIAI research agentsInference Optimization
0 likes · 22 min read
AI That Builds AI: Naive AI's Self-Evolving Models Beat Top Benchmarks in 151 Experiments
AntTech
AntTech
Sep 29, 2026 · Industry Insights

Token Efficiency: Key Variable for Green Computing in the AI Era

Ant Group and CAICT's joint report defines token efficiency across four dimensions—intelligence output, economics, experience, and energy-carbon—and proposes a four-layer technical framework from model to application layers, validated by SME and medical case studies showing 2-4x cost reduction and 4.5x efficiency gains, plus a five-dimensional measurement system for quantifiable green computing.

AI Energy ConsumptionCarbon EmissionsIndustry Report
0 likes · 14 min read
Token Efficiency: Key Variable for Green Computing in the AI Era
AntData
AntData
Sep 29, 2026 · Industry Insights

Token Efficiency: The Critical Lever for Sustainable AI Computing

A joint research report by Ant Group and CAICT defines Token efficiency across four dimensions, presents a four-layer technical framework, analyzes two real-world cases showing 60-80% Token cost reduction and 4.5x efficiency gains, and proposes a five-dimensional measurement system to quantify green computing value.

AI infrastructureInference Optimizationagent architecture
0 likes · 12 min read
Token Efficiency: The Critical Lever for Sustainable AI Computing
Huajiao Technology
Huajiao Technology
Sep 23, 2026 · Artificial Intelligence

H3 Max Speed Teardown: How Co-Design and Inference Optimization Achieve Faster-Than-Real-Time Video Generation

The article dissects H3 Max's 2.46-second generation of 5-second 768p video, attributing speed to three layers: post-training that reduces sampling steps without quality loss, a co-designed inference stack (Falcon engine) lowering per-step cost, and GB200 hardware; it clarifies inference time vs end-to-end latency, and explains latent refinement versus traditional super-resolution.

AI infrastructureH3 MaxInference Optimization
0 likes · 21 min read
H3 Max Speed Teardown: How Co-Design and Inference Optimization Achieve Faster-Than-Real-Time Video Generation
Top Architecture Tech Stack
Top Architecture Tech Stack
Sep 20, 2026 · Artificial Intelligence

GLM-5.3-FlashX Hits 200 tokens/s: Agent-Driven Dense Feedback Loops Optimize Full Inference Pipeline

Zhipu AI's GLM-5.3-FlashX achieves 200 tokens/s, a 5x speedup over Flash, built on 100k domestic chips using Agent-driven dense feedback loops that optimize the full inference pipeline through three case studies: FP32/TF32 precision fixes, GIL contention in KV Transfer, and redundant computation elimination in KDA Decode.

Agent-driven optimizationFP32/TF32 precisionGIL contention
0 likes · 13 min read
GLM-5.3-FlashX Hits 200 tokens/s: Agent-Driven Dense Feedback Loops Optimize Full Inference Pipeline
Machine Heart
Machine Heart
Sep 3, 2026 · Artificial Intelligence

Apple's IVT Internalizes Visual Reasoning for 5x Faster Proactive Video Prediction

Apple researchers propose Internalized Visual Thinking (IVT), a post-training framework that learns predictive world modeling during training but removes future-frame generation at inference, achieving 5x speedup over Visual CoT while matching or exceeding accuracy on Early-event and Next-event Prediction across Ego-Exo4D, Ego4D, and EPIC-KITCHENS-100.

Apple ResearchFlux-VAEInference Optimization
0 likes · 11 min read
Apple's IVT Internalizes Visual Reasoning for 5x Faster Proactive Video Prediction
Tencent Technical Engineering
Tencent Technical Engineering
Aug 28, 2026 · Artificial Intelligence

Tencent Hunyuan Hy4 Preview: 770B Open-Source Model Claims Top Tier

Tencent releases Hunyuan Hy4 preview, a 770B parameter mixture-of-experts model with 49B activated parameters and 1M context length, achieving top-tier open-source performance across coding, office, gaming, and scientific tasks, with internal blind tests showing 2.99/4 score surpassing GLM 5.3 and Kimi K3, plus 31.8% inference throughput gains via self-optimization.

HunyuanHy4Inference Optimization
0 likes · 6 min read
Tencent Hunyuan Hy4 Preview: 770B Open-Source Model Claims Top Tier
TechVision Expert Circle
TechVision Expert Circle
Aug 17, 2026 · Artificial Intelligence

Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market

In early 2026, open-weight LLMs such as Llama 4, Qwen 3 and DeepSeek‑V3/R2 began matching or surpassing leading closed models like GPT‑4.5 and Claude Opus 5, driven by MoE architectures, FP8 precision, GRPO training and aggressive inference optimizations, prompting a reassessment of enterprise AI strategy.

AI economicsInference OptimizationOpen Source LLM
0 likes · 15 min read
Open-Source LLMs Close the Gap: Low-Cost AI Poised to Redefine the Market
Machine Heart
Machine Heart
Aug 14, 2026 · Artificial Intelligence

PhyAI: The First Unified Edge‑Cloud Inference Runtime for Physical AI

PhyAI introduces a unified inference runtime that serves four Physical AI deployment scenarios—benchmark, cloud RL rollout, edge, and factory MaaS—by consolidating model code, employing a Model Runner and Scheduler, and using a Control‑Time Roofline analysis to reveal latency bottlenecks, achieving up to 4.65× speedup while highlighting the joint limits of hardware and environment on robot control frequency.

Control‑Time RooflineEdge‑Cloud InferenceInference Optimization
0 likes · 8 min read
PhyAI: The First Unified Edge‑Cloud Inference Runtime for Physical AI
Smart Era Software Development
Smart Era Software Development
Jul 30, 2026 · R&D Management

Token Costs Double Monthly, Yet 70% of AI Projects Just Please the Boss

Despite an 80% token price drop and a 20‑fold surge in global token usage, many enterprises face exploding monthly token bills, with 70% of AI projects merely placating managers; the article analyzes token economics, efficiency breakthroughs, and a task‑centric workflow that can halve costs and boost productivity.

AI Project EfficiencyAI cost managementInference Optimization
0 likes · 15 min read
Token Costs Double Monthly, Yet 70% of AI Projects Just Please the Boss
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Jul 28, 2026 · Artificial Intelligence

Day0 Adaptation of Kimi K3 on Alibaba Cloud Lingjun Zhenwu M890 Supernode

On July 27, Alibaba Cloud announced that its Lingjun Zhenwu M890 supernode instance has been successfully adapted to run the 2.8‑trillion‑parameter Kimi K3 model, achieving a 35 % reduction in first‑token latency, a 1.8× increase in decode throughput, and support for up to 1 M token context through joint chip, software‑stack and framework optimizations.

Inference OptimizationKDAKimi K3
0 likes · 6 min read
Day0 Adaptation of Kimi K3 on Alibaba Cloud Lingjun Zhenwu M890 Supernode
ThinkingAgent
ThinkingAgent
Jul 25, 2026 · Artificial Intelligence

From Next Token to Deployable AI: A Comprehensive Overview of Large Model Technology

This article maps the entire large‑model production chain—from data collection, token prediction, and architecture design through training, alignment, inference, multimodal perception, agentic action, deployment, evaluation, and safety—highlighting key engineering decisions, trade‑offs, and concrete examples.

Agent SafetyInference OptimizationRetrieval-Augmented Generation
0 likes · 47 min read
From Next Token to Deployable AI: A Comprehensive Overview of Large Model Technology
Meituan Technology Team
Meituan Technology Team
Jul 9, 2026 · Artificial Intelligence

LongCat-2.0 Open‑Sourced: Meituan Releases Domestic‑Chip Optimized Inference Code

Meituan has open‑sourced LongCat-2.0, a 1.6‑trillion‑parameter model that runs inference on domestic GPU/NPUs through sparse attention, N‑gram embedding, and a MOPD architecture, providing multi‑precision weights and deployment scripts to enable stable, high‑throughput serving on existing Chinese hardware.

Inference OptimizationLongCat-2.0N-gram embedding
0 likes · 7 min read
LongCat-2.0 Open‑Sourced: Meituan Releases Domestic‑Chip Optimized Inference Code
Wu Shixiong's Large Model Academy
Wu Shixiong's Large Model Academy
Jul 3, 2026 · Artificial Intelligence

Deep Research Series: 12 Articles From the Basic Loop to the First Training Review

This article reorganizes a 12‑part Deep Research Agent series into a logical learning path, summarizing each part’s problem, key solutions, and practical takeaways—from building a runnable loop and handling tool failures to data construction, context management, and training evaluation.

Context ManagementDeep ResearchInference Optimization
0 likes · 12 min read
Deep Research Series: 12 Articles From the Basic Loop to the First Training Review
Design Hub
Design Hub
Jun 29, 2026 · Artificial Intelligence

When AI Starts Getting Real Work Done, Are We Ready to Evaluate It?

The article analyzes recent AI updates—from DeepSeek's DSpark inference boost and FlashAttention‑4's kernel redesign to Codex UI tweaks and design‑mode tools—arguing that the competition is shifting from answering questions to actually completing tasks, and it highlights three layers of progress, evaluation challenges, and the practical questions we must now ask of AI agents.

AIAgent AutomationDeepSeek
0 likes · 19 min read
When AI Starts Getting Real Work Done, Are We Ready to Evaluate It?
Raymond Ops
Raymond Ops
Jun 28, 2026 · Operations

Why Large‑Model Services Keep Running Out of GPU Memory: An Ops View from KV Cache to Concurrency

The article explains why large‑model inference services frequently hit GPU memory limits, breaks down static vs. dynamic memory consumption, shows how KV‑Cache, request length, and concurrency amplify usage, and provides a step‑by‑step troubleshooting and mitigation workflow for production environments.

GPU memoryInference OptimizationKV Cache
0 likes · 26 min read
Why Large‑Model Services Keep Running Out of GPU Memory: An Ops View from KV Cache to Concurrency
21CTO
21CTO
Jun 27, 2026 · Artificial Intelligence

Large vs Small Language Models: An Apple‑Centric Technical Comparison

The article analyses how deployment targets, inference economics, and training budgets drive divergent design choices for large (LLM) and small (SLM) Transformer‑based language models, covering architecture tweaks, data‑centric training methods, quantisation, KV‑cache management, and hybrid routing strategies for production systems.

Hybrid InferenceInference Optimizationlarge language models
0 likes · 16 min read
Large vs Small Language Models: An Apple‑Centric Technical Comparison
AI Engineer Programming
AI Engineer Programming
Jun 8, 2026 · Artificial Intelligence

When to Use Small Models: A System Design Perspective

Small models are chosen based on deployment constraints rather than absolute parameter counts; the article outlines how resource limits, latency, cost, privacy, and task characteristics define their suitability, compares their strengths and weaknesses to large models, and offers system‑level design patterns for effective use.

Inference OptimizationLLM DeploymentRAG
0 likes · 20 min read
When to Use Small Models: A System Design Perspective
Old Zhang's AI Learning
Old Zhang's AI Learning
May 31, 2026 · Artificial Intelligence

vLLM 0.22 Release: Production-Ready DeepSeek V4 and Extreme KV Cache Compression

The vLLM 0.22 stable release introduces production‑grade DeepSeek V4 support, massive kernel fusions, up to 10‑20× speedups, Batch Invariance with 28.9% latency gain, a Rust front‑end, multi‑level KV cache offload that can double context length, and broad hardware coverage across NVIDIA, AMD, CPU and RISC‑V, making it a pivotal upgrade for inference infrastructure teams.

Batch InvarianceInference OptimizationKV Cache
0 likes · 13 min read
vLLM 0.22 Release: Production-Ready DeepSeek V4 and Extreme KV Cache Compression
Xiaomi Tech
Xiaomi Tech
May 30, 2026 · Artificial Intelligence

How Xiaomi’s MiMo V2.5 Achieves 99% API Price Cut with Full‑Stack Inference Optimizations

The MiMo‑V2.5 series combines Hybrid Sliding‑Window Attention, Mixture‑of‑Experts and multimodal support with a complete redesign of KVCache management, tiered caching, prefix‑tree logic and scheduling, compressing KVCache to about one‑seventh of full‑attention models and delivering up to 40% faster Prefill, 30% lower TTFT and dramatically reduced inference costs that enable a 99% API price reduction.

Hybrid SWAInference OptimizationKVCache
0 likes · 12 min read
How Xiaomi’s MiMo V2.5 Achieves 99% API Price Cut with Full‑Stack Inference Optimizations
Tencent Technical Engineering
Tencent Technical Engineering
May 24, 2026 · Artificial Intelligence

How Tsinghua & Tencent Mixed‑X Won the MLSys 2026 MoE Inference Challenge with a 4.1× Speedup

The Tsinghua‑Tencent Mixed‑X team captured the MLSys 2026 MoE inference optimization championship by analyzing NPU bottlenecks, redesigning data movement, applying expert‑level sharding, continuous DMA, PSUM batching, and an Agent‑based optimizer, achieving a 4.1× end‑to‑end speedup while preserving bit‑level output fidelity.

Agent optimizerInference OptimizationMLSys 2026
0 likes · 14 min read
How Tsinghua & Tencent Mixed‑X Won the MLSys 2026 MoE Inference Challenge with a 4.1× Speedup
ZhiKe AI
ZhiKe AI
May 23, 2026 · Artificial Intelligence

Zhipu AI Unveils GLM-5.1-HighSpeed, Achieving 400 Tokens/s and 6× Faster Generation

On May 22 2026, Zhipu AI released the GLM‑5.1‑HighSpeed variant, which generates up to 400 tokens per second—over six times the speed of the standard GLM‑5.1 and twice that of Google’s Gemini‑3.5‑Flash—thanks to multi‑dimensional inference, attention and sequence‑parallel optimizations while preserving full model capabilities.

GLM-5.1-HighSpeedInference OptimizationLLM
0 likes · 3 min read
Zhipu AI Unveils GLM-5.1-HighSpeed, Achieving 400 Tokens/s and 6× Faster Generation
SuanNi
SuanNi
May 22, 2026 · Artificial Intelligence

How GLM‑5.1‑highspeed Achieves 7× Faster Inference to Become the World’s Fastest Flagship Model

On May 22, Zhipu launched the GLM‑5.1‑highspeed API, delivering 400 tokens per second—about 7× faster than the original model and twice as fast as Gemini 3.5 Flash—through a three‑layer optimization that rewrites the MoE inference path, introduces dynamic scheduling, and leverages TileRT’s AOT engine to cut latency while preserving full flagship capabilities.

GLM-5.1Inference OptimizationTileRT
0 likes · 10 min read
How GLM‑5.1‑highspeed Achieves 7× Faster Inference to Become the World’s Fastest Flagship Model
IT Services Circle
IT Services Circle
May 17, 2026 · Artificial Intelligence

60 Essential AI Terms Every Programmer Should Master

This article walks programmers through 60 core AI concepts—from the basics of large language models and tokens to advanced topics like prompt engineering, retrieval‑augmented generation, fine‑tuning, and inference optimization—organized into progressive skill levels and illustrated with concrete examples and code snippets.

AIInference OptimizationRAG
0 likes · 25 min read
60 Essential AI Terms Every Programmer Should Master
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 14, 2026 · Artificial Intelligence

Elastic Speculative Decoding Breaks Large‑Model Inference Bottlenecks

The paper introduces ECHO, an elastic speculative decoding framework that treats token verification as a global budget‑scheduling problem, uses sparse confidence gating and a two‑level priority scheduler, and demonstrates up to 14.4% throughput gains for high‑concurrency LLM serving.

Inference Optimizationelastic budgetlarge language models
0 likes · 14 min read
Elastic Speculative Decoding Breaks Large‑Model Inference Bottlenecks
Machine Heart
Machine Heart
May 11, 2026 · Artificial Intelligence

How PRISM Enables Efficient Test‑Time Scaling for Discrete Diffusion Language Models

The article analyzes how the PRISM framework redesigns test‑time scaling for discrete diffusion language models by replacing costly Best‑of‑N sampling with a three‑stage hierarchical search, local branching via partial remasking, and self‑verified feedback, achieving large accuracy gains on math and code benchmarks while cutting inference compute by up to four‑fold.

Discrete DiffusionHierarchical SearchInference Optimization
0 likes · 11 min read
How PRISM Enables Efficient Test‑Time Scaling for Discrete Diffusion Language Models
Lao Guo's Learning Space
Lao Guo's Learning Space
May 7, 2026 · Artificial Intelligence

Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference

The article explains why large‑language‑model inference is bottlenecked by memory‑bandwidth, then details Google’s Gemma 4 MTP technique—using a small draft model with speculative decoding and shared KV‑Cache—to parallelize token prediction, achieving up to three‑fold speed gains without any loss in output quality, and provides step‑by‑step local deployment instructions.

Gemma 4Inference OptimizationKV Cache
0 likes · 11 min read
Gemma 4 MTP Deep Dive: Speculative Decoding & KV‑Cache Sharing for 3× Faster Inference
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
May 2, 2026 · Artificial Intelligence

RouteMoA: Dynamic Routing Without Pre‑Inference for Efficient Multi‑Agent Mixtures

RouteMoA moves model selection ahead of inference by using a lightweight scorer to predict each model's suitability from the query, dramatically cutting computation cost and latency while preserving or improving accuracy, as demonstrated on a 15‑model pool with up to 90% cost reduction and 64% latency reduction.

ACL 2026Inference OptimizationMixture of Agents
0 likes · 9 min read
RouteMoA: Dynamic Routing Without Pre‑Inference for Efficient Multi‑Agent Mixtures
PaperAgent
PaperAgent
Apr 29, 2026 · Artificial Intelligence

Skill‑Driven Reasoning Cuts Tokens by Up to 59% While Boosting Accuracy

The article introduces the TRS (Thinking with Reasoning Skills) framework, which distills historical LLM reasoning traces into reusable skill cards, enabling offline skill‑base construction and online retrieval that dramatically reduces token consumption (6‑59%) and often improves accuracy on math and coding tasks.

Inference OptimizationMath ReasoningReasoning Skills
0 likes · 13 min read
Skill‑Driven Reasoning Cuts Tokens by Up to 59% While Boosting Accuracy
Machine Heart
Machine Heart
Apr 25, 2026 · Artificial Intelligence

Can Multi-Model Co-Evolution Shatter the Single-Model Ceiling? Squeeze Evolve Achieves Validator-Free SOTA Inference

The paper introduces Squeeze Evolve, a validator‑free multi‑model evolutionary framework that orchestrates diverse large language models to break the performance ceiling of any single model, delivering up to 23‑point accuracy improvements and 1.4‑3.3× cost reductions across math, vision, and scientific benchmarks.

AI researchInference OptimizationSqueeze Evolve
0 likes · 8 min read
Can Multi-Model Co-Evolution Shatter the Single-Model Ceiling? Squeeze Evolve Achieves Validator-Free SOTA Inference
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Apr 21, 2026 · Artificial Intelligence

Can Linear Attention Complete Prefill-as-a-Service for Cross‑Datacenter Heterogeneous PD Separation?

The article analyzes why the massive KVCache bandwidth required by heterogeneous pre‑fill/ decode (PD) separation cannot be solved at the system level, proposes a Prefill‑as‑a‑Service architecture that leverages linear‑attention models to cut KVCache generation, and validates the design with a 1‑trillion‑parameter Kimi Linear deployment that achieves 54% higher throughput and 64% lower P90 TTFT across a 100 Gbps inter‑datacenter link.

Heterogeneous PDInference OptimizationKVCache
0 likes · 7 min read
Can Linear Attention Complete Prefill-as-a-Service for Cross‑Datacenter Heterogeneous PD Separation?
Geek Labs
Geek Labs
Apr 20, 2026 · Artificial Intelligence

A Complete Open‑Source Guide to LLM Internals: From Tokenization to Inference Optimization

This open‑source tutorial breaks down large language model internals into 11 detailed topics—covering BPE tokenization, attention mathematics, backpropagation, transformer architecture, KV‑Cache, Paged and Flash Attention, and frontier techniques—each with numeric derivations and Python code, making it ideal for developers and interview preparation.

AttentionFlash AttentionInference Optimization
0 likes · 5 min read
A Complete Open‑Source Guide to LLM Internals: From Tokenization to Inference Optimization
Xiaomi Tech
Xiaomi Tech
Apr 10, 2026 · Artificial Intelligence

Xiaomi AI’s 8× Faster Mobile Inference and OCR‑Free 80‑Page Document Understanding at ACL 2026

Xiaomi’s AI team announced seven ACL 2026 papers that span low‑bit KV‑cache quantization for 8.3× faster LLM inference, OCR‑free multi‑page document VQA, a new attention‑basin analysis, non‑autoregressive spoken dialogue generation, a comprehensive mobile‑agent benchmark, a success‑rate‑aware training policy, and a progressive universal information‑extraction framework.

Document UnderstandingInference Optimizationbenchmark
0 likes · 12 min read
Xiaomi AI’s 8× Faster Mobile Inference and OCR‑Free 80‑Page Document Understanding at ACL 2026
AI Tech Publishing
AI Tech Publishing
Apr 5, 2026 · Artificial Intelligence

Why the First Token Is Slow: A Deep Dive into KV Cache for LLM Inference

The article explains how KV cache eliminates redundant computations in autoregressive LLM generation, detailing the attention mechanism, the O(n²) waste of recomputing K and V, the cache‑based solution, its impact on time‑to‑first‑token, and the memory‑vs‑speed trade‑off.

AttentionInference OptimizationKV Cache
0 likes · 7 min read
Why the First Token Is Slow: A Deep Dive into KV Cache for LLM Inference
DeepHub IMBA
DeepHub IMBA
Apr 2, 2026 · Artificial Intelligence

Speculative Decoding Explained: Small Draft Model + One‑Shot Verification

The article details how speculative decoding—using a fast small model to draft tokens and a large model to verify them—overcomes the memory‑bandwidth bottleneck of autoregressive inference, introduces SSD’s self‑draft and tree‑verification stages, presents real‑world benchmark gains, and shows how to enable it in vLLM.

GPU memory bandwidthInference OptimizationSSD
0 likes · 14 min read
Speculative Decoding Explained: Small Draft Model + One‑Shot Verification
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 28, 2026 · Artificial Intelligence

GigaWorld-Policy Boosts Inference Speed 10× and Success Rate 30%

The newly released GigaWorld-Policy world‑action model replaces traditional video‑prediction‑heavy WAM designs with an action‑centered architecture, achieving a ten‑fold inference speedup, ten‑fold training efficiency gain, and a 30% increase in real‑robot task success rate while reducing memory usage compared with Motus and Cosmos‑Policy.

Action-Centered ArchitectureInference Optimizationmultimodal learning
0 likes · 8 min read
GigaWorld-Policy Boosts Inference Speed 10× and Success Rate 30%
Machine Learning Algorithms & Natural Language Processing
Machine Learning Algorithms & Natural Language Processing
Mar 7, 2026 · Artificial Intelligence

How Princeton’s DYSCO Decoder Boosts Long-Context Reasoning by 25% Without Fine‑Tuning

The DYSCO (Dynamic Attention‑Scaling Decoding) algorithm, introduced by Princeton’s Chen Danqi team and NYU, eliminates the need for fine‑tuning and restores performance on long‑context tasks, delivering up to a 25% relative gain on 128K token benchmarks while adding only about 3.8% extra FLOPs.

DYSCODynamic Attention ScalingInference Optimization
0 likes · 10 min read
How Princeton’s DYSCO Decoder Boosts Long-Context Reasoning by 25% Without Fine‑Tuning
Alibaba International Intelligent Technology
Alibaba International Intelligent Technology
Mar 5, 2026 · Artificial Intelligence

Engineering Exploration of Generative Recall: Leveraging Large Models for Next‑Gen Recommendation Retrieval

The article analyzes the limitations of traditional ID‑based and vector recall, proposes a generative Semantic‑ID‑based recall framework (ASGR), details a series of inference‑time optimizations—including a custom SDPA‑enabled optimizer, novel beam search, static‑cache compilation, tree‑aware attention, sliding‑window attention, quantization, ONNX/Optimum conversion, and TensorRT‑LLM integration—and reports concrete latency and throughput gains that enable large‑scale, low‑latency recommendation serving.

Inference OptimizationRecommendation Systembeam search
0 likes · 25 min read
Engineering Exploration of Generative Recall: Leveraging Large Models for Next‑Gen Recommendation Retrieval
SuanNi
SuanNi
Feb 27, 2026 · Artificial Intelligence

How Dual‑Channel Loading Doubles LLM Inference Throughput

The article analyzes the storage‑bandwidth bottleneck of agent‑style large language models, explains why traditional pre‑fill and decode architectures underutilize network resources, and details a dual‑channel loading and smart scheduling design that unlocks idle bandwidth, achieving up to 1.9× higher throughput in both offline and online inference workloads.

AI infrastructureDual-Channel LoadingInference Optimization
0 likes · 14 min read
How Dual‑Channel Loading Doubles LLM Inference Throughput
Data Party THU
Data Party THU
Feb 25, 2026 · Artificial Intelligence

Why Multimodal LLMs Miss Tiny Objects—and How to Fix It

This article analyzes why multimodal large language models often fail to detect small objects, identifies three core bottlenecks, and presents a four‑tiered optimization roadmap—from zero‑cost inference tricks to data augmentation, model fine‑tuning, and engineering safeguards—backed by three real‑world case studies and actionable guidelines.

Data AugmentationInference Optimizationmodel fine-tuning
0 likes · 20 min read
Why Multimodal LLMs Miss Tiny Objects—and How to Fix It
AI Engineering
AI Engineering
Feb 16, 2026 · Artificial Intelligence

Qwen3.5-397B: 397B‑Parameter Multimodal LLM Boosts Inference Speed 8‑19×

Alibaba’s Qwen3.5-397B-A17B, a 397‑billion‑parameter open‑source multimodal LLM, combines mixed linear attention with a sparse MoE architecture to achieve 8.6‑19× higher decoding throughput than Qwen3‑Max, supports 201 languages, and can be deployed via vLLM, Docker, Transformers, or SGLang with various optimization presets.

Inference OptimizationQwen3.5large language model
0 likes · 8 min read
Qwen3.5-397B: 397B‑Parameter Multimodal LLM Boosts Inference Speed 8‑19×
DeWu Technology
DeWu Technology
Feb 11, 2026 · Artificial Intelligence

How Generative Models Transform Re‑ranking Architecture for Faster, More Diverse Recommendations

This article examines the evolution of re‑ranking systems from traditional pointwise models to a two‑stage generation‑evaluation framework, compares autoregressive and non‑autoregressive generative approaches, details inference speed optimizations with GPU and model‑server upgrades, and outlines a future end‑to‑end sequence generation architecture enhanced by reinforcement learning and contrastive learning.

AIGenerative ModelsInference Optimization
0 likes · 14 min read
How Generative Models Transform Re‑ranking Architecture for Faster, More Diverse Recommendations
Tencent Technical Engineering
Tencent Technical Engineering
Jan 30, 2026 · Artificial Intelligence

Can Rendering Thought Chains as Images Speed Up LLM Reasoning?

This article introduces Render‑of‑Thought (RoT), a novel paradigm that compresses chain‑of‑thought reasoning into visual embeddings using frozen vision encoders, achieving 3‑4× token reduction, faster inference, and improved interpretability while requiring minimal pre‑training.

Inference Optimizationchain-of-thoughtlatent space
0 likes · 12 min read
Can Rendering Thought Chains as Images Speed Up LLM Reasoning?
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 26, 2026 · Artificial Intelligence

How We Scaled a 3.5B MoE LLM for Real‑Time Search Relevance

This article details the engineering challenges and solutions for deploying a 3.5 billion‑parameter MoE LLM in Taobao's search relevance pipeline, covering large‑batch scheduling, dynamic load balancing, intra‑batch KV‑Cache reuse, and MoE kernel tuning to meet sub‑second latency requirements.

Inference OptimizationKV CacheLLM
0 likes · 15 min read
How We Scaled a 3.5B MoE LLM for Real‑Time Search Relevance
Code Archaeology
Code Archaeology
Jan 19, 2026 · Artificial Intelligence

LLM Inference Optimization: Metrics, Model & Service Techniques

This article systematically covers LLM inference optimization across performance metrics (TTFT, TPOT, throughput, goodput, MFU, MBU), model-level techniques (quantization, distillation, pruning, speculative decoding, attention optimizations), and service-level strategies (continuous batching, prefill/decode separation, prompt caching, parallelism) to improve latency, throughput, and cost-efficiency.

Inference OptimizationKV CacheLLM inference
0 likes · 14 min read
LLM Inference Optimization: Metrics, Model & Service Techniques
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 23, 2025 · Artificial Intelligence

How Hybrid Transformer‑Mamba Architectures Overcome KVCache Challenges in Large‑Model Inference

This article explains how SGLang’s hybrid model design combines Transformer attention with Mamba state‑space layers, introduces a dual‑pool memory architecture and elastic allocation, and presents specialized prefix‑cache and speculative‑decoding techniques that together enable efficient, scalable inference for long‑context large language models.

Hybrid ModelsInference OptimizationKVCache
0 likes · 22 min read
How Hybrid Transformer‑Mamba Architectures Overcome KVCache Challenges in Large‑Model Inference
MaGe Linux Operations
MaGe Linux Operations
Dec 19, 2025 · Artificial Intelligence

Boost vLLM Inference Throughput by 40% with Three Simple Config Tweaks

After discovering that only a few vLLM settings truly impact performance, this guide details how adjusting gpu_memory_utilization, max_num_batched_tokens, and enabling chunked prefill can raise Qwen2.5‑72B‑Instruct throughput from ~1800 to over 2500 tokens/s, improve latency, and provides comprehensive deployment, monitoring, and troubleshooting instructions.

DockerGPUInference Optimization
0 likes · 30 min read
Boost vLLM Inference Throughput by 40% with Three Simple Config Tweaks
Tencent Cloud Developer
Tencent Cloud Developer
Dec 9, 2025 · Artificial Intelligence

How Do Large Language Models Turn Text into Math? A Deep Dive into Transformers

This article walks through the complete workflow of AI large language models, from turning user queries into token matrices via tokenization and embedding, through the Transformer’s self‑attention and multi‑head mechanisms, to decoding logits into human‑readable text, while also covering position encoding, long‑context strategies, generation parameters, and practical engineering tips.

Inference OptimizationSelf-AttentionTransformer
0 likes · 29 min read
How Do Large Language Models Turn Text into Math? A Deep Dive into Transformers
Data Party THU
Data Party THU
Oct 21, 2025 · Artificial Intelligence

Can Linear‑Time LSTMs Beat Transformers? Scaling Laws Reveal the Answer

The paper presents a systematic scaling‑law study of the linear‑time xLSTM architecture versus quadratic‑time Transformers, evaluating parameter‑data loss surfaces, optimal model size under equal FLOP budgets, and inference latency components, and shows that xLSTM consistently offers better cost‑effectiveness across diverse contexts and budgets.

Inference OptimizationLinear Time ComplexityModel Efficiency
0 likes · 11 min read
Can Linear‑Time LSTMs Beat Transformers? Scaling Laws Reveal the Answer
AntTech
AntTech
Oct 13, 2025 · Artificial Intelligence

How dInfer Accelerates Diffusion LLM Inference Over 10× Faster Than Fast‑dLLM

Ant Group's open‑source dInfer framework dramatically speeds up diffusion language model inference—achieving more than a ten‑fold boost over Fast‑dLLM, surpassing autoregressive baselines, and delivering 1011 tokens per second on HumanEval—by tackling computational cost, KV‑cache invalidation, and parallel decoding challenges through modular system‑level innovations.

AI performanceInference OptimizationLLM
0 likes · 11 min read
How dInfer Accelerates Diffusion LLM Inference Over 10× Faster Than Fast‑dLLM
DataFunSummit
DataFunSummit
Oct 8, 2025 · Artificial Intelligence

How EasyRec Boosts Recommendation Training and Inference Performance

This article explains the EasyRec recommendation system’s training and inference architecture, detailing optimization techniques such as embedding parallelism, CPU/GPU placement, XLA and TRT fusion, online learning pipelines, network compression, and real‑world deployment results that dramatically improve throughput and latency.

AI infrastructureEasyRecInference Optimization
0 likes · 15 min read
How EasyRec Boosts Recommendation Training and Inference Performance
AntTech
AntTech
Sep 14, 2025 · Artificial Intelligence

Ring-mini-2.0: How a 16B MoE Model Delivers 128K Context and 500+ Tokens/s

Ring-mini-2.0 is a high‑performance inference MoE model that activates only 1.4 B parameters out of 16 B total, achieving dense‑model quality below 10 B while supporting 128 K context length and ultra‑fast generation speeds of over 300 tokens/s.

AIInference OptimizationMoE
0 likes · 4 min read
Ring-mini-2.0: How a 16B MoE Model Delivers 128K Context and 500+ Tokens/s
DataFunSummit
DataFunSummit
Sep 11, 2025 · Artificial Intelligence

How Meituan’s MTGR is Redefining Generative Recommendation at Scale

This article explains why Meituan introduced a generative recommendation model, describes the MTGR architecture, data organization, training and inference engines built on TorchRec and TensorRT, reports performance gains and cost reductions, and outlines future directions such as simplifying the recommendation funnel and cross‑business heterogeneous modeling.

Inference OptimizationMTGRScaling Law
0 likes · 15 min read
How Meituan’s MTGR is Redefining Generative Recommendation at Scale
Eric Tech Circle
Eric Tech Circle
Sep 10, 2025 · Artificial Intelligence

Deploy High‑Performance Local LLMs with vLLM: A Step‑by‑Step Guide

This article walks through installing and configuring vLLM for local large language model inference, compares it with Ollama and LM Studio, details environment setup, model download, testing scripts, and shows how to expose an OpenAI‑compatible API for production use.

Inference OptimizationModelScopeOpenAI API
0 likes · 11 min read
Deploy High‑Performance Local LLMs with vLLM: A Step‑by‑Step Guide
Baobao Algorithm Notes
Baobao Algorithm Notes
Sep 2, 2025 · Artificial Intelligence

How LongCat‑Flash Achieves Record Speed and Efficiency for a 560B MoE Model

LongCat‑Flash is a 560‑billion‑parameter Mixture‑of‑Experts LLM that combines a dynamic zero‑computation expert design, shortcut‑connected MoE communication, variance‑aligned scaling, and a three‑stage agent‑centric pre‑training pipeline, delivering over 100 TPS on H800 GPUs at a cost of $0.70 per million tokens.

Inference OptimizationLongCat-FlashTraining Infrastructure
0 likes · 23 min read
How LongCat‑Flash Achieves Record Speed and Efficiency for a 560B MoE Model
Architects' Tech Alliance
Architects' Tech Alliance
Aug 18, 2025 · Artificial Intelligence

How Large Model Training Dominates Compute and What New Techniques Can Change It

This article explains why pre‑training large AI models consumes 90‑99% of total compute, describes the full training and inference pipelines, introduces resource‑saving strategies such as PD‑separation, and reviews market trends and infrastructure challenges shaping the next generation of AI systems.

AI TrainingAI infrastructureGPU architecture
0 likes · 13 min read
How Large Model Training Dominates Compute and What New Techniques Can Change It
AIWalker
AIWalker
Aug 4, 2025 · Artificial Intelligence

Can Lumina-mGPT 2.0 Replace Diffusion Models? A Deep Dive into Its Autoregressive Power

Lumina-mGPT 2.0 is a decoder‑only, zero‑shot trained autoregressive image model that rivals diffusion systems like DALL·E 3 in quality while offering unified multimodal tokenization, flexible multi‑task generation, and several inference‑speed tricks, yet it still faces licensing, scaling and sampling‑time challenges.

AI model analysisImage GenerationInference Optimization
0 likes · 22 min read
Can Lumina-mGPT 2.0 Replace Diffusion Models? A Deep Dive into Its Autoregressive Power
Tencent Technical Engineering
Tencent Technical Engineering
Jul 18, 2025 · Artificial Intelligence

From CPUs to GPUs: How Traditional Backend Skills Power Modern AI Infrastructure

This article explores the evolution of AI infrastructure, comparing it with traditional backend systems, and details how hardware shifts to GPU-centric designs, software adaptations like deep learning frameworks, and engineering challenges in model training and inference can be addressed using established backend methodologies.

AI infrastructureGPU computingInference Optimization
0 likes · 19 min read
From CPUs to GPUs: How Traditional Backend Skills Power Modern AI Infrastructure
Tencent Technical Engineering
Tencent Technical Engineering
Jul 11, 2025 · Artificial Intelligence

How DeepSeek Achieved 15,800+ Tokens/s: Full‑Stack Inference Optimizations

This article details the Angel‑HCF team's end‑to‑end DeepSeek inference optimizations—including PD separation, multi‑layer MTP, EP and DP parallelism, hardware‑aware kernels, and load‑balancing strategies—that boost throughput to over 15,800 tokens per second while keeping per‑token latency under 50 ms.

AI performanceDeepSeekGPU utilization
0 likes · 13 min read
How DeepSeek Achieved 15,800+ Tokens/s: Full‑Stack Inference Optimizations
Alibaba Cloud Developer
Alibaba Cloud Developer
Jun 10, 2025 · Artificial Intelligence

How AI Application Architectures Evolve: From Simple LLM Calls to Guardrails, Routing, and Agents

This article traces the evolution of AI application architectures—from the earliest minimal user‑LLM interaction to advanced designs featuring context enhancement, input/output guardrails, intent routing, model gateways, caching strategies, agent capabilities, monitoring, and inference performance optimizations—providing practical insights and references for developers.

AI ArchitectureAgentCaching
0 likes · 21 min read
How AI Application Architectures Evolve: From Simple LLM Calls to Guardrails, Routing, and Agents
Meituan Technology Team
Meituan Technology Team
May 15, 2025 · Artificial Intelligence

How Meituan’s MTGR Framework Achieved 65× Faster Inference with Scaling Laws

Meituan’s recommendation team introduced the MTGR framework, aligning traditional DLRM features with a unified HSTU‑based Transformer to explore scaling laws, delivering a 65‑fold FLOPs boost, 12% lower inference cost, and record gains in online CTR and order volume across its food‑delivery platform.

Inference OptimizationMTGRScaling Law
0 likes · 26 min read
How Meituan’s MTGR Framework Achieved 65× Faster Inference with Scaling Laws
Baidu Geek Talk
Baidu Geek Talk
May 12, 2025 · Artificial Intelligence

One‑Click Deployment of Baidu Qwen3 Large Models on Baidu Baige AI Platform

This guide explains how to use Baidu Baige's AI heterogeneous computing platform to deploy the eight‑model Qwen3 family—including dense and MoE variants—via a one‑click process, covering resource configuration, inference acceleration options, and post‑deployment service access.

AIBaidu BaigeInference Optimization
0 likes · 4 min read
One‑Click Deployment of Baidu Qwen3 Large Models on Baidu Baige AI Platform
AI Algorithm Path
AI Algorithm Path
May 1, 2025 · Artificial Intelligence

Uncovering the Secrets of LLM Inference Optimization

This article dissects the major bottlenecks of large‑language‑model serving—prefill vs. decode, sparsity, memory bandwidth, KV‑cache growth—and walks through concrete engineering tricks such as paged attention, radix‑tree KV caches, compressed attention, speculative decoding, FlexGen weight scheduling, FastServe queuing, plus a runnable vLLM code snippet.

FastServeFlexGenInference Optimization
0 likes · 18 min read
Uncovering the Secrets of LLM Inference Optimization
JD Retail Technology
JD Retail Technology
Apr 22, 2025 · Artificial Intelligence

Generative Large‑Model Architecture for JD Advertising: Practices, Challenges, and Optimization

JD’s advertising platform replaces rule‑based recall with a generative large‑model pipeline that unifies e‑commerce knowledge, multimodal user intent, and semantic IDs across recall, coarse‑ranking, fine‑ranking and creative optimization, while meeting sub‑100 ms latency and sub‑¥1‑per‑million‑token cost through quantization, parallelism, caching, and joint generative‑discriminative inference, delivering double‑digit performance gains and paving the way for domain‑specific foundation models.

AdvertisingGenerative AIInference Optimization
0 likes · 20 min read
Generative Large‑Model Architecture for JD Advertising: Practices, Challenges, and Optimization
Architects' Tech Alliance
Architects' Tech Alliance
Apr 13, 2025 · Artificial Intelligence

Deploying DeepSeek LLMs On-Premises: Step‑by‑Step Guide and Hardware Sizing

This article provides a comprehensive technical guide for privately deploying DeepSeek large language models, covering model and runtime parameter selection, hardware sizing calculations, software stack preparation, inference service setup, performance tuning, and security monitoring considerations.

AI hardware sizingDeepSeekInference Optimization
0 likes · 14 min read
Deploying DeepSeek LLMs On-Premises: Step‑by‑Step Guide and Hardware Sizing
Ops Development & AI Practice
Ops Development & AI Practice
Mar 19, 2025 · Artificial Intelligence

Can Cache‑Augmented Generation Outperform RAG? A Deep Dive into LLM Efficiency

Cache‑augmented generation (CAG) preloads documents into LLM context using KV caches to eliminate retrieval latency, offering faster inference for static knowledge bases, while RAG remains more flexible for dynamic or large corpora; this article compares their definitions, performance, implementation steps, and future prospects.

CAGCache AugmentationInference Optimization
0 likes · 11 min read
Can Cache‑Augmented Generation Outperform RAG? A Deep Dive into LLM Efficiency
Baidu Tech Salon
Baidu Tech Salon
Mar 13, 2025 · Artificial Intelligence

How PaddlePaddle 3.0 Boosts Large‑Model Inference with 4‑Bit Quantization and MLA Optimizations

PaddlePaddle 3.0 introduces a full‑stack inference engine that supports FP8, INT8, and 4‑bit quantization for popular LLMs such as DeepSeek V3/R1, delivers up to 2× token throughput on a single H800 GPU, and provides detailed deployment scripts for single‑node and multi‑node setups, including MTP speculative decoding and SageAttention for long‑sequence acceleration.

DockerInference OptimizationMLA
0 likes · 13 min read
How PaddlePaddle 3.0 Boosts Large‑Model Inference with 4‑Bit Quantization and MLA Optimizations
Java Architecture Diary
Java Architecture Diary
Mar 7, 2025 · Artificial Intelligence

Boost Inference Efficiency with QwQ-32B: Benchmarks, Resource Savings, and Java Integration

QwQ-32B, Alibaba’s new inference‑optimized large language model built on the Qwen2.5 architecture, outperforms DeepSeek‑R1 across math reasoning, code generation, and safety benchmarks while requiring only 24 GB vRAM, and the article provides detailed performance data, resource‑efficiency analysis, and step‑by‑step Java and Ollama integration instructions.

Function CallingInference OptimizationJava integration
0 likes · 7 min read
Boost Inference Efficiency with QwQ-32B: Benchmarks, Resource Savings, and Java Integration
Architects' Tech Alliance
Architects' Tech Alliance
Feb 27, 2025 · Artificial Intelligence

How Inspur Metabrain R1 Server Enables 1000+ Concurrent Users for DeepSeek 671B via SGLang Optimization

The Inspur Metabrain R1 inference server, equipped with FP8 acceleration and a 1128 GB HBM3e memory pool, has been tightly integrated with SGLang 0.4.3 to run the 671‑billion‑parameter DeepSeek R1 model, delivering over 1,000 concurrent user sessions and up to 3,976 tokens/s throughput.

AI serverDeepSeekInference Optimization
0 likes · 5 min read
How Inspur Metabrain R1 Server Enables 1000+ Concurrent Users for DeepSeek 671B via SGLang Optimization
Architects' Tech Alliance
Architects' Tech Alliance
Feb 12, 2025 · Industry Insights

DeepSeek’s Technical Innovations: MoE Architecture, Efficient Inference, and Multimodal Capabilities

The article analyzes DeepSeek’s recent breakthroughs—including its Mixture‑of‑Experts architecture, cost‑effective inference optimizations, high‑accuracy multimodal processing, and open‑source collaboration—while also offering a curated bundle of technical e‑books covering AI chips, networking, storage, and more.

DeepSeekInference Optimizationartificial intelligence
0 likes · 4 min read
DeepSeek’s Technical Innovations: MoE Architecture, Efficient Inference, and Multimodal Capabilities
DeWu Technology
DeWu Technology
Feb 12, 2025 · Artificial Intelligence

Edge Intelligence for Intelligent Video Cover Recommendation

The article describes an edge‑based video‑cover recommendation system for DeWu that leverages the MNN SDK and a lightweight MobileNetV3 model, performing on‑device inference with quantization and parallel processing to automatically select high‑quality covers, achieving sub‑second latency and boosting click‑through rates by up to 18 %.

Inference OptimizationVideo Coveredge AI
0 likes · 12 min read
Edge Intelligence for Intelligent Video Cover Recommendation
JD Retail Technology
JD Retail Technology
Feb 12, 2025 · Artificial Intelligence

Accelerating Generative Recommendation with NVIDIA TensorRT‑LLM in JD Advertising

JD Advertising accelerates its generative‑recall recommendation system by integrating NVIDIA TensorRT‑LLM, which simplifies the pipeline, injects LLM knowledge, scales to billions of parameters, and delivers over five‑fold throughput gains, one‑fifth the cost, and significant CTR improvements in both recommendation and search.

Inference OptimizationLLMTensorRT-LLM
0 likes · 13 min read
Accelerating Generative Recommendation with NVIDIA TensorRT‑LLM in JD Advertising
DataFunTalk
DataFunTalk
Jan 26, 2025 · Artificial Intelligence

58.com’s LingXi Large Language Model Platform: Development, Deployment, and Performance Optimizations

Since the launch of ChatGPT, 58.com has built a Model‑as‑a‑Service platform called LingXi that trains and serves domain‑specific large language models, supports over a hundred internal scenarios with daily inference exceeding ten million calls, and continuously improves performance through quantization, GPU optimization, model miniaturization, and advanced AI applications such as interview assistants, voice agents, and RAG‑enabled agents.

AI PlatformAI applicationsInference Optimization
0 likes · 9 min read
58.com’s LingXi Large Language Model Platform: Development, Deployment, and Performance Optimizations
JD Tech Talk
JD Tech Talk
Jan 14, 2025 · Artificial Intelligence

Advantages and Engineering Implementation of Generative Recommendation Systems Using Large Language Models

This article explains how generative recommendation systems powered by large language models simplify the recommendation pipeline, integrate world knowledge, benefit from scaling laws, and require specialized engineering optimizations such as TensorRT‑LLM deployment, inference acceleration, and hybrid model strategies to achieve low latency and high throughput in real‑world e‑commerce scenarios.

AIInference OptimizationLLM
0 likes · 10 min read
Advantages and Engineering Implementation of Generative Recommendation Systems Using Large Language Models
Baobao Algorithm Notes
Baobao Algorithm Notes
Jan 3, 2025 · Artificial Intelligence

How DeepSeek-V3 Achieves Massive Scale with FP8, MoE, and System Optimizations

The article examines DeepSeek‑V3’s architecture and training pipeline, highlighting its use of MLA and a highly granular MoE design, pioneering FP8 mixed‑precision training, fine‑grained per‑tile quantization, advanced parallelism strategies, and inference optimizations such as PD separation and NanoFlow to achieve unprecedented efficiency on limited GPU resources.

DeepSeek-V3FP8Inference Optimization
0 likes · 10 min read
How DeepSeek-V3 Achieves Massive Scale with FP8, MoE, and System Optimizations
DataFunSummit
DataFunSummit
Dec 31, 2024 · Artificial Intelligence

How Momo Leverages Large Model Technology to Transform Business and R&D Processes

This article explains how Momo utilizes large language model technologies to revamp its AI application paradigm, achieve efficient inference through quantization and prefix caching, build a workflow‑based model platform, and outline future plans for framework optimization and multimodal support.

AI PlatformInference OptimizationMomo
0 likes · 16 min read
How Momo Leverages Large Model Technology to Transform Business and R&D Processes
DataFunSummit
DataFunSummit
Nov 22, 2024 · Artificial Intelligence

EasyRec Recommendation Algorithm Training and Inference Optimization

This article presents a comprehensive overview of EasyRec’s recommendation system architecture, detailing training and inference optimizations, embedding parallelism, CPU/GPU placement strategies, online learning pipelines, and network compression techniques that together improve scalability, latency, and cost efficiency.

EasyRecInference Optimizationdistributed systems
0 likes · 15 min read
EasyRec Recommendation Algorithm Training and Inference Optimization
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Sep 26, 2024 · Artificial Intelligence

How Alibaba Cloud’s PAI Tackles Large‑Model Training and Inference Challenges in 2024

At the 2024 Yunqi Conference, Alibaba Cloud’s AI Infra experts detailed the latest challenges of large‑model deployment—such as hardware costs, resource management, and software‑hardware coordination—and introduced PAI’s new capabilities, including stability tools, automated distributed training, reinforcement‑learning frameworks, inference optimizations, and integrated big‑data AI solutions.

AI InfraInference Optimizationbig-data-integration
0 likes · 14 min read
How Alibaba Cloud’s PAI Tackles Large‑Model Training and Inference Challenges in 2024
Sohu Tech Products
Sohu Tech Products
Aug 28, 2024 · Artificial Intelligence

EasyRec Recommendation Algorithm Training and Inference Optimization

EasyRec, Alibaba Cloud’s modular recommendation framework, unifies configurable data, embedding, dense, and output layers on MaxCompute, EMR, and DLC, and speeds training with deduplication, EmbeddingParallel sharding, lock‑free hash tables, GPU embeddings, and AMX BF16, while inference benefits from operator fusion, low‑precision AVX/AMX kernels, compact caches, batch merging, and network compression, enabling real‑time online learning and delivering higher recommendation quality at lower cost in e‑commerce.

Alibaba CloudEasyRecInference Optimization
0 likes · 14 min read
EasyRec Recommendation Algorithm Training and Inference Optimization
DataFunTalk
DataFunTalk
Aug 26, 2024 · Artificial Intelligence

EasyRec Recommendation Algorithm Training and Inference Optimization

This article presents a comprehensive overview of EasyRec's recommendation system architecture, detailing training and inference optimizations, distributed deployment strategies, operator fusion techniques, online learning pipelines, and network-level improvements to enhance performance and scalability.

AIInference Optimizationrecommendation
0 likes · 15 min read
EasyRec Recommendation Algorithm Training and Inference Optimization
Baobao Algorithm Notes
Baobao Algorithm Notes
Aug 26, 2024 · Artificial Intelligence

Master Essential LLM Engineering Skills: Transform, Model, and Infer with Custom Scripts

This guide presents a hands‑on curriculum of core large‑model engineering tasks—including model conversion scripts, custom modeling wrappers, multi‑model inference utilities, and channel‑aware loss tracking—to help practitioners build practical, reusable tools without deep theoretical overhead.

AI EngineeringInference OptimizationPython Scripting
0 likes · 8 min read
Master Essential LLM Engineering Skills: Transform, Model, and Infer with Custom Scripts