Tagged articles
online inference
22 articles · Page 1 of 1
This article presents the LaD model, an end‑to‑end generative and detoxification framework for search query auto‑completion that hierarchically captures long‑ and short‑term user interests, achieves large AB test gains on Kuaishou, and has been accepted at KDD 2025.
Generative ModelsKDD2025LaD
0 likes · 11 min read

This article details the design and evolution of DHPS, Kuaishou's load‑balanced, RDMA‑based high‑performance service architecture, explaining its network, storage, and traffic‑scheduling innovations that deliver over 270% query‑throughput improvement, lower latency, reduced CPU usage, and near‑five‑nine availability for large‑scale AI inference workloads.
RDMAdistributed systemshigh-performance computing
0 likes · 17 min read

The document describes the evolution, design principles, key technologies, online inference workflow, evaluation methods, and sample‑generation techniques of a large‑language‑model‑based multi‑agent system that powers a 24/7 e‑commerce merchant assistant, highlighting its benefits, challenges, and future work.
AI planningLLMmulti-agent
0 likes · 21 min read

This article explains how large language models reshape recommendation systems by simplifying pipelines, integrating world knowledge, and leveraging scaling laws, and details the engineering steps for deploying generative recall models—including product encoding, user prompting, model training, TensorRT‑LLM optimization, and continuous performance improvements.
LLMTensorRT-LLMai-optimization
0 likes · 13 min read

This report details how large language models (LLMs) were integrated into Tencent's search advertising pipeline—from early extraction‑distillation experiments in 2023 to a 2024 end‑to‑end generative recall architecture—showing significant improvements in relevance, diversity, and revenue through knowledge injection, supervised fine‑tuning, constrained beam‑search decoding, and high‑performance inference services.
AILLMbeam search
0 likes · 11 min read

The paper describes how Meituan’s food‑delivery search advertising uses a heterogeneous billion‑node graph and the GraphET engine to boost weak‑supply recall, detailing a progression from fine‑grained modeling to GPT‑enhanced pre‑training, and presenting a scalable training and low‑latency inference architecture that handles hundreds of billions of edges.
GraphETLarge-Scale GraphMeituan
0 likes · 27 min read

This article describes how JD Retail's advertising technology team tackled the high‑compute demands of modern recommendation models by designing a distributed graph‑partitioned heterogeneous computing framework, introducing TensorBatch request aggregation, leveraging deep‑learning compiler bucketing and asynchronous compilation, and implementing a multi‑stream GPU architecture to dramatically improve online inference throughput and latency.
Deep Learning CompilerGPU accelerationdistributed computing
0 likes · 13 min read

To cope with soaring traffic on Bilibili’s effect‑advertising engine, the team systematically measured latency, eliminated redundant Redis calls, switched JSON to Protobuf, applied branch‑prediction hints, loop‑unrolling and AVX256 SIMD, introduced object‑pooling and an inverted‑index request format, cutting CPU usage by 21 % and boosting peak throughput 13 %.
C++CPUmemory management
0 likes · 21 min read

This article demonstrates how to build a complete end-to-end machine-learning workflow for taxi trip duration prediction by integrating OpenMLDB with Alibaba Cloud MaxCompute’s serverless services, covering environment setup, offline data ingestion, feature extraction, model training, deployment, and real-time online inference within 20 ms.
Feature StoreMaxComputeOpenMLDB
0 likes · 13 min read

This article describes the architecture, key features, dynamic deployment, performance optimizations, and real‑world results of a high‑throughput online inference platform that serves deep‑learning models for JD.com’s risk‑control decision engine, achieving near‑hundred‑fold latency improvements.
AIModel ServingRisk Control
0 likes · 11 min read

The article reviews the development of NetEase Leihuo's game recommendation system, covering the shift from offline batch recommendation to real‑time feature engineering and online inference, detailing architecture design, practical experiences, performance optimizations, and future directions such as real‑time training.
AIgame industryonline inference
0 likes · 8 min read

Evaluating the Cloud Music online estimation system on NUMA‑based servers revealed that CPU pinning across both memory nodes dramatically boosts throughput on high‑end 96‑core machines—up to 75% for complex models—while low‑end servers gain only modestly, confirming NUMA‑aware scheduling’s critical role for CPU‑intensive inference workloads.
CPU architectureNUMAonline inference
0 likes · 8 min read

The Dual Vector Foil (DVF) system decouples index construction from model training by building a post‑training HNSW graph, enabling any complex model to score candidates, which yields a 5.7 % recall boost, cuts latency from ~40 ms to 6.5 ms, and raises QPS over tenfold while simplifying maintenance.
Large-Scale Retrievaldual vector foilindexing
0 likes · 27 min read

This article describes the challenges of deploying a large BERT model as an online service for 360 Search and details engineering optimizations—including framework selection, model quantization, knowledge distillation, stream scheduling, caching, and dynamic sequence handling—that dramatically improve latency, throughput, and resource utilization.
BERTFP16 quantizationGPU optimization
0 likes · 12 min read

This article details the engineering challenges of serving a large BERT model in real‑time for 360 Search and describes a series of optimizations—including TensorRT‑based kernel fusion, model quantization, knowledge distillation, multi‑stream execution, caching, and dynamic sequence handling—that together achieve low latency, high throughput, and stable deployment on GPU clusters.
BERTGPUTensorRT
0 likes · 10 min read

iQIYI’s short‑video recommendation team replaced a GBDT coarse‑ranking model with a lightweight dual‑tower DNN, applied knowledge distillation, sparse‑aware embedding optimization, and inference merging, then introduced a cascade MMOE architecture, achieving comparable accuracy with half the memory, ~19 ms latency reduction, and measurable gains in watch time, CTR and engagement.
Recommendation Systemcascade modelcoarse ranking
0 likes · 15 min read

This article examines the challenges of moving machine‑learning models from offline training to online serving, proposes a modular architecture—including model gateway, data source gateway, business service center, monitoring, and RPC components—to enable rapid model deployment, version management, traffic mirroring, gray‑release, and real‑time monitoring.
Model Servingmachine learningmonitoring
0 likes · 10 min read

This article explains how AI technologies are transitioning from labs to the web, covering neural network fundamentals, the distinction between cloud and edge intelligence, implementation pipelines, offline model optimization, online inference backends like WebGL and WASM, and practical web front‑end AI use cases.
Web AIfrontendmachine learning
0 likes · 10 min read

This article presents a comprehensive overview of Ctrip's hotel recommendation system, covering its technical architecture, data processing pipelines, various ranking and embedding models—including FM, Wide&Deep, DeepFM, and FTRL—deployment methods such as PMML and TensorFlow Serving, offline and online evaluation results, and challenges like cold‑start and diversity.
CtripEmbeddingRecommendation System
0 likes · 24 min read

Meituan’s AI‑driven search combines the Augur online inference framework—offering stateless, distributed feature operators, transformers, and a DSL for rapid, high‑throughput model scoring—with the Poker platform for model training, versioning, and experimentation, together accelerating iteration, improving performance, and enabling advanced model‑as‑feature ensembles.
AI PlatformModel Servingfeature engineering
0 likes · 26 min read


58 TechDec 20, 2019 · Artificial Intelligence
The article presents a comprehensive overview of 58.com’s AI platform built on Kubernetes, detailing its layered architecture, resource scheduling, offline training pipelines, debugging environment, distributed TensorFlow/PyTorch training, performance benchmarks, and online inference services, highlighting how the system empowers various business units with scalable AI capabilities.
KubernetesPyTorchTensorFlow
0 likes · 11 min read

This article explains how to construct a high‑elastic, high‑performance end‑to‑end federated learning pipeline—including task scheduling, visual modeling, model management, version control, and online inference—using the FATE‑Flow platform to move from experimental ML to production deployment.
AIFATE-Flowmodel management
0 likes · 14 min read
