Tagged articles
GPU Inference
25 articles · Page 1 of 1

AntTechOct 15, 2026 · Artificial Intelligence
Ant Group's Bailin team open-sources Ling-3.1-flash, a 560B-parameter mixture-of-experts model that activates 25B parameters per token, supports 1M context, and achieves 210 tokens/sec output speed with an intelligence index of 41 on Artificial Analysis, while quantization reduces weight from 534GB to 292GB enabling deployment on 4 Hopper GPUs instead of 8.
GPU InferenceLing-3.1-flashMixture of Experts
0 likes · 11 min read

This article details a zero-downtime model update strategy using immutable artifacts, a model registry, canary deployments with readiness probes, traffic shifting via service mesh, and optional in-process generational hot loading, demonstrated with a GPU ranking service handling 600 QPS and 7 GiB models.
GPU InferenceIstioKubernetes
0 likes · 32 min read

Alibaba's TaoLive AIGC team open-sources TaoMate-H3, a streaming audio-video generation model based on MiniMax H3 that uses three-step LoRA inference and autoregressive generation to produce minute-long videos with synchronized dialogue, singing, and sound effects at multiple resolutions, achieving 11-27x speedups on H20 GPUs.
GPU InferenceLoRAMiniMax H3
0 likes · 17 min read

The article details how iQIYI migrated its ad conversion‑rate (CVR) model from a TensorFlow‑CPU pipeline to a TorchRec‑based PyTorch‑GPU architecture, expanding parameters tenfold while aligning offline AUC and online performance, and outlines the systematic optimizations that yielded both business metric gains and cost reductions.
CVR modelGPU InferencePyTorch
0 likes · 22 min read

The vLLM 0.20.1 patch, released shortly after 0.20.0, consolidates stability fixes and performance optimizations for DeepSeek V4, adds several bug fixes, updates installation instructions, and provides targeted upgrade recommendations for different user scenarios.
GPU Inferencebug fixdeepseek-v4
0 likes · 9 min read

The article details how Claude Opus 4.6's chain‑of‑thought data were used to distill the 27‑billion‑parameter Qwen3.5‑27B model with Unsloth and LoRA, achieving full‑context inference on a single RTX 3090/4090, while outlining performance numbers, hyper‑parameter tips, benchmark gains and the trade‑offs of losing multimodal abilities.
Claude Opus 4.6GPU InferenceLoRA
0 likes · 7 min read

PaddleOCR‑VL‑1.5, the latest Baidu release, uses only 0.9 B parameters to achieve 94.5% accuracy on OmniDocBench v1.5, surpassing larger open‑source and commercial OCR models, while offering multi‑task, multi‑language support, lightweight deployment, and detailed performance benchmarks.
DeepSeek-OCRGPU InferenceMulti-language
0 likes · 9 min read


58 TechJan 6, 2026 · Artificial Intelligence
This article provides a step‑by‑step technical walkthrough of vLLM 0.8.4 on a single GPU, detailing the platform’s startup, model loading, Multi‑LoRA deployment, internal ZMQ communication, request scheduling, and inference execution, while exposing key source‑code snippets and architectural diagrams.
GPU InferenceLoRA adaptersModel Serving
0 likes · 35 min read

Over ten years Pinterest transformed a fragmented machine‑learning stack into a unified AI platform, iterating through stages from early ad‑hoc pipelines to scalable GPU‑accelerated services, while learning that timing, organization alignment, and efficiency are crucial for lasting impact.
AI PlatformGPU InferenceInfrastructure
0 likes · 25 min read

This article examines the high cost and low throughput of embedding vectors in log‑processing scenarios, analyzes the performance bottlenecks of inference frameworks, and details a series of cloud‑native optimizations—including switching to vLLM, deploying multiple model replicas with Triton, decoupling tokenization, and priority queuing—that together raise throughput by 16× and reduce per‑token pricing by two orders of magnitude.
EmbeddingGPU InferenceVector Index
0 likes · 9 min read

A recent migration of a multimodal image inference system from an internal network to a cloud environment revealed that NVLink bridges dramatically improve multi‑GPU inference speed by reducing inter‑GPU communication overhead, while tensor‑parallel and data‑parallel strategies each have distinct trade‑offs for model deployment.
AI performanceData ParallelGPU Inference
0 likes · 11 min read

This article walks through TensorRT's core optimization features, auxiliary debugging tools, and a step‑by‑step SMPLer‑X case study, showing how graph simplification, mixed‑precision, and engine generation cut inference latency to roughly 22‑29% of the original runtime.
GPU InferenceModel AccelerationONNX
0 likes · 6 min read

This article provides a comprehensive tutorial on locally deploying the 671‑billion‑parameter DeepSeek R1 model using Ollama, covering model selection, hardware requirements, dynamic quantization, detailed installation steps, performance observations, and practical recommendations for consumer‑grade hardware.
AI Model OptimizationDeepSeekDynamic Quantization
0 likes · 14 min read

An in‑depth analysis shows that NVLink 3.0 reduces all‑reduce communication latency for Llama 3 70B inference from over 1.8 seconds to under 100 ms, delivering a dramatic speedup compared with PCIe 4.0 and highlighting the critical role of high‑bandwidth interconnects in large‑model deployments.
All-reduceGPU InferenceLlama-3
0 likes · 5 min read

This article details JD Retail's ad‑tech team's deep‑compute optimizations—including a distributed graph‑based heterogeneous framework, GPU‑focused inference engine enhancements, TensorBatch request aggregation, deep‑learning compiler bucket pre‑compilation, asynchronous compilation, and multi‑stream GPU processing—to overcome high‑concurrency, low‑latency online recommendation challenges.
Deep Learning CompilerGPU Inferencedistributed computing
0 likes · 14 min read

This guide explains how to set up Alibaba Cloud's ACK environment, install the Cloud Native AI Suite, configure TensorRT, and run Stable Diffusion with dramatically reduced latency and memory usage, including detailed commands, performance metrics, and reproducible code snippets.
AI accelerationGPU InferenceStable Diffusion
0 likes · 7 min read

The article chronicles Baidu Search’s Model Architecture Group’s evolution of deep‑learning‑driven search, detailing the shift from inverted‑index to semantic vector indexing, the use of transformer‑based models for text and image queries, large‑scale offline/online pipelines, and extensive GPU‑centric optimizations such as pruning, quantization and distillation, all aimed at delivering precise, cost‑effective results to hundreds of millions of users.
ERNIEGPU InferenceSearch Architecture
0 likes · 14 min read

This article compiles Meituan's cutting‑edge computer‑vision research and engineering achievements—including CVPR award‑winning segmentation, YOLOv6 releases, GPU inference optimizations, the Food2K dataset, and numerous paper digests—to provide practical insights for visual AI practitioners.
CVPRComputer VisionFood2K
0 likes · 11 min read

The article outlines practical GPU inference optimization for NetEase’s news recommendation, covering model analysis with Netron, multi‑GPU parallelism, memory‑copy reduction, batch sizing, TensorRT conversion and tuning, custom plugins, and the GRPS serving framework to achieve significant latency and utilization gains.
GPU InferenceTensorRTmodel optimization
0 likes · 44 min read

This article details how the metaapp advertising team adopted Alibaba Cloud's open‑source DeepRec to overcome parameter‑server bottlenecks, compress terabyte‑scale embeddings, leverage GPU‑accelerated distributed training, and build a low‑maintenance, high‑performance inference service using DeepRec's Processor and oneDNN optimizations.
DeepRecEmbeddingVariableGPU Inference
0 likes · 13 min read

By isolating CPU preprocessing and post‑processing from GPU inference into separate processes and applying TensorRT’s FP16/INT8 optimizations, the custom Python framework boosts Python vision inference services from roughly 4.5 to 27.4 QPS—a 5‑10× speedup—while reducing GPU utilization and cost.
CPU-GPU SeparationCUDAGPU Inference
0 likes · 14 min read

Vivo’s AI recommendation team leveraged Alibaba’s DeepRec engine—introducing dynamic Embedding Variables, feature admission/elimination, Parquet datasets, and advanced CPU/GPU inference optimizations such as SessionGroup, device placement, multi‑stream and BladeDISC compilation—resulting in notable gains in model accuracy, latency reduction, and resource efficiency.
DeepRecGPU Inferencedynamic embedding
0 likes · 13 min read

This article explains why traditional CPU inference and naïve GPU usage are inefficient for recommendation workloads, introduces NVIDIA Multi‑Process Service (MPS) technology, describes VIVO's custom Rust‑based inference engine and deployment strategies, and presents performance and cost benefits along with practical deployment considerations.
GPU InferenceKubernetesMPS
0 likes · 13 min read

TurboTransformers introduces a suite of GPU‑centric optimizations—including a high‑throughput batch reduction algorithm, a variable‑length‑aware memory allocator, and a dynamic‑programming‑based batch scheduling strategy—that together deliver significantly lower latency and higher throughput for Transformer‑based NLP services compared with existing frameworks such as PyTorch, TensorFlow, ONNX Runtime and TensorRT.
BERTGPU InferenceTurboTransformers
0 likes · 13 min read


58 TechNov 6, 2019 · Artificial Intelligence
This article explains how the 58 AI platform leverages NVIDIA TensorRT to accelerate deep‑learning inference on GPUs, describes three integration approaches, details the TF‑TRT implementation and Kubernetes deployment, and presents performance gains for ResNet‑50 and OCR models.
AI PlatformGPU InferenceKubernetes Deployment
0 likes · 7 min read
