Tagged articles
Large Model Inference
15 articles · Page 1 of 1

JD TechJun 25, 2026 · Artificial Intelligence
JD donated its self‑developed Oxygen xLLM large‑model inference engine to the OpenAtom Open Source Foundation under Apache 2.0, highlighting its service‑engine decoupled architecture, heterogeneous‑chip support, proven performance gains in e‑commerce, power and public‑safety use cases, and a roadmap to become the domestic AI‑infra standard.
AI infrastructureLarge Model InferenceOpenAtom
0 likes · 9 min read

JD announced the donation of its Oxygen xLLM inference engine to the OpenAtom Open‑Source Foundation, detailing its service‑engine decoupled architecture, performance breakthroughs across e‑commerce, power and public‑safety workloads, and a roadmap to expand the open‑source AI ecosystem.
AI infrastructureEngineering IntelligenceLarge Model Inference
0 likes · 8 min read

On June 24, 2026 JD announced the donation of its Oxygen xLLM large‑model inference engine to the OpenAtom Open Source Foundation, detailing its service‑engine decoupled architecture, performance breakthroughs, heterogeneous chip support, and real‑world gains in e‑commerce, power‑grid and public‑safety applications while outlining a roadmap for broader ecosystem co‑building and standards leadership.
AI infrastructureEngineering IntelligenceLarge Model Inference
0 likes · 7 min read

The article details how Huolala’s Dolphin platform engineers large‑model inference for high‑concurrency, long‑context, low‑latency production workloads, achieving 50‑60% GPU cost reduction through systematic resource allocation, model quantization, PD‑separation, speculative sampling, and kernel‑level optimizations while maintaining service stability.
GPU utilizationLarge Model InferenceResource Optimization
0 likes · 18 min read

The article outlines ten actionable digital trends for 2026—from AI agents and large‑model inference cost strategies to platform engineering maturity, FinOps‑driven multi‑cloud governance, zero‑trust deepening, edge AI deployment, data contracts, and value‑centric tech organization—explaining why each matters and how CTOs can quickly build competence.
AI agentData ContractFinOps
0 likes · 15 min read

This article details how Baidu Cloud's hybrid‑cloud team leveraged the vLLM framework to cut the cold‑start time of massive models like Qwen3‑235B‑A22B from minutes to a few seconds through accelerated weight loading, CUDA‑graph capture postponement, cross‑instance state reuse, fork‑based process startup, and guard‑instance pre‑warming techniques.
CUDA GraphLarge Model Inferencecold-start optimization
0 likes · 16 min read

This article details Baidu's Tianchi supernode design and software tuning—covering hardware scale‑up, deployment planning, Prefill and Decode stage optimizations, quantization strategies, and communication schemes—to dramatically boost large‑model inference throughput and latency while lowering token‑cost.
AI infrastructureLarge Model InferenceParallelism
0 likes · 20 min read

Mooncake, an open‑source large‑model inference platform, introduces a KVCache‑centric architecture that dramatically improves throughput, reduces latency and cuts inference costs by up to 20%, while integrating with frameworks like SGLang and vLLM and leveraging Alibaba Cloud’s eRDMA and GPUDirect technologies for scalable, high‑performance deployments.
AI performanceAlibaba CloudKVCache
0 likes · 7 min read

The article reviews high‑performance inference strategies for large language models such as Deepseek‑R1, detailing CPU‑GPU process separation, Paged and Radix Attention, Chunked Prefill, output‑length reduction, tensor‑parallel multi‑GPU scaling, and speculative decoding, each shown to markedly boost throughput and cut latency in real deployments.
AIGPU accelerationLarge Model Inference
0 likes · 22 min read

Large‑model inference engines convert prompts into responses via a Prefill stage and an autoregressive Decoder, measured by TTFT and TPOT, and Baidu’s AIAK suite improves TPOT by separating tokenization, using static slot scheduling, and asynchronous execution, cutting token‑interval latency from ~35 ms to ~14 ms and boosting GPU utilization to about 75 % while also leveraging quantization and speculative execution for higher throughput.
AI accelerationGPU utilizationLarge Model Inference
0 likes · 10 min read

This article explains the architecture of large‑model inference engines, key performance metrics like TTFT and TPOT, the limitations of popular engines such as vLLM, and Baidu Baige's AIAK solutions—including multi‑process, static slot, and asynchronous execution—that dramatically reduce token‑interval latency and increase GPU utilization.
AIAKGPU utilizationLLM performance
0 likes · 10 min read

This talk presents the Ant Group team's recent work on large‑model inference memory optimization, covering GPU memory challenges, virtual memory management (VMM), the Virtual Tensor framework, LayerKV techniques, performance comparisons with Page Attention and FlashAttention, and extensive experimental results demonstrating reduced latency and higher QPS.
AttentionGPULarge Model Inference
0 likes · 25 min read

In June 2024, Alibaba Cloud and Tsinghua University's MADSys Lab announced the open‑source Mooncake architecture, a KVCache‑centered large‑model inference framework that boosts throughput, lowers cost, and standardizes resource‑pooling techniques for high‑performance AI inference across industry and academia.
KVCacheLarge Model InferenceTsinghua University
0 likes · 4 min read

Large-model offline (batch) inference, which processes massive data on billion-parameter models, faces GPU memory and distributed scheduling challenges; this article explains how Ray's cloud-native framework, model parallelism, and Ray Datasets pipelines address these issues, improve throughput, and enable elastic, efficient GPU utilization.
GPU utilizationLarge Model InferenceRay
0 likes · 16 min read

This article analyzes Baidu’s ERNIE‑ViLG 2.0 and PaddlePaddle‑optimized Stable Diffusion models, presenting benchmark comparisons, hardware‑specific speed and memory gains, and the underlying inference optimizations that enable low‑cost, high‑throughput AI‑generated image creation.
AI paintingAIGCGPU acceleration
0 likes · 9 min read
