Tagged articles
Model Serving
42 articles · Page 1 of 1
The article explains two deployment patterns for integrating machine learning models into Palantir Foundry's Ontology—real-time function calls and batch inference—so predictions become regular fields on business objects, enabling business users to consume AI outputs without technical expertise, while highlighting governance practices like versioning, confidence intervals, and wrapper functions for safe AI agent usage.
AI governanceBatch InferenceMachine Learning Deployment
0 likes · 10 min read

This article details a zero-downtime model update strategy using immutable artifacts, a model registry, canary deployments with readiness probes, traffic shifting via service mesh, and optional in-process generational hot loading, demonstrated with a GPU ranking service handling 600 QPS and 7 GiB models.
GPU InferenceIstioKubernetes
0 likes · 32 min read

NVIDIA's DGX Cloud Home toolkit links consumer RTX 40/50 GPUs into a distributed cluster for LLM inference and fine-tuning, using pipeline and tensor parallelism; benchmarks show three RTX 5070s achieving near-A100 throughput for Llama 3.1 70B, though network bandwidth and stability remain key limitations.
Llama 3.1 70BModel ServingNVIDIA DGX Cloud Home
0 likes · 14 min read

This article details the architecture and optimization of a Prediction Server for fine-ranking in recommender systems, covering model management, batch inference, multi-objective fusion, probability calibration, deployment pipelines, and high-availability patterns to score thousands of candidates within milliseconds.
Batch InferenceHigh AvailabilityModel Serving
0 likes · 46 min read

The article analyzes why the massive KVCache bandwidth required by heterogeneous pre‑fill/ decode (PD) separation cannot be solved at the system level, proposes a Prefill‑as‑a‑Service architecture that leverages linear‑attention models to cut KVCache generation, and validates the design with a 1‑trillion‑parameter Kimi Linear deployment that achieves 54% higher throughput and 64% lower P90 TTFT across a 100 Gbps inter‑datacenter link.
Heterogeneous PDInference OptimizationKVCache
0 likes · 7 min read


58 TechJan 6, 2026 · Artificial Intelligence
This article provides a step‑by‑step technical walkthrough of vLLM 0.8.4 on a single GPU, detailing the platform’s startup, model loading, Multi‑LoRA deployment, internal ZMQ communication, request scheduling, and inference execution, while exposing key source‑code snippets and architectural diagrams.
GPU InferenceLoRA adaptersModel Serving
0 likes · 35 min read

This article compares Ray Serve and Celery, explaining their design philosophies, scaling models, GPU‑aware scheduling, operational trade‑offs, and real‑world case studies to help engineers choose the right tool for high‑throughput online inference or large‑scale batch processing.
GPUModel ServingRay Serve
0 likes · 9 min read

This guide walks you through installing Ollama, pulling the open‑source DeepSeek‑R1:1.5B model, wrapping it with a Go‑based Model Context Protocol (MCP) server, creating a client example, and enhancing the experience with Open‑WebUI while offering performance‑tuning tips.
DeepSeekGoMCP
0 likes · 9 min read

An in‑depth analysis reveals TensorFlow’s rapid decline, the rise of PyTorch, and how Ant Group’s OpenRank‑driven “Large Model Open‑Source Ecosystem Panorama 2.0” maps shifting trends, from short‑term hype projects to performance‑focused AI infrastructure, highlighting the emerging US‑China dominance in AI open‑source development.
AI ecosystemAI open-sourceModel Serving
0 likes · 15 min read

This article provides a comprehensive side‑by‑side comparison of the open‑source LLM serving tools Xinference and Ollama, examining their core goals, architecture, model support, deployment options, performance, ecosystem integration, typical use cases, future roadmap, and guidance on selecting the right solution for enterprise or personal projects.
LLMModel Servingcomparison
0 likes · 7 min read

Baidu Cloud's Qianfan platform launched DeepSeek‑R1 and DeepSeek‑V3 with ultra‑low inference pricing, leveraging advanced engine performance tweaks, a split Prefill/Decode architecture, and comprehensive security measures that together boost throughput, cut costs, and ensure enterprise‑grade reliability.
AI inferenceBaidu CloudModel Serving
0 likes · 5 min read

This guide explains how to deploy a production‑ready DeepSeek‑R1 inference service on Alibaba Cloud ACK using KServe, covering model preparation, storage configuration, service deployment, observability, autoscaling, model acceleration, gray‑release and GPU‑shared inference.
DeepSeekGPUKServe
0 likes · 13 min read

This article details a comprehensive engineering practice for optimizing AI inference services at ZhiZhuan, covering background analysis, selection of TorchServe over alternatives, GPU/CPU performance tuning, custom handlers, Torch‑TRT integration, and deployment on Kubernetes, with measured improvements in throughput and resource utilization.
AI inferenceGPU optimizationKubernetes
0 likes · 16 min read

iQIYI’s minute‑level online deep‑learning framework overcomes stability, timeliness, compatibility, delayed feedback, catastrophic forgetting, and i.i.d. constraints through high‑availability pipelines, TensorFlow Example serialization, rapid P2P model distribution, flexible scheduling, disaster‑recovery rollbacks, PU‑loss adjustment, and knowledge‑distillation, delivering a 6.2% revenue boost.
AdvertisingCTR predictionModel Serving
0 likes · 9 min read

This article details Xiaohongshu's end‑to‑end GPU‑based transformation of its recommendation and search models, covering background, model characteristics, training and inference frameworks, system‑level and GPU‑level optimizations, compilation tricks, hardware upgrades, and future directions for large‑scale machine‑learning infrastructure.
GPUModel Servinglarge-scale systems
0 likes · 18 min read

Meituan’s Fulfillment Platform team built a high‑performance distributed causal‑forest framework—named Causal On Spark—that trains hundreds of trees on hundreds of millions of samples within minutes using MapReduce‑based histogram splitting, extensive memory optimizations, Parquet model serving, and novel distributed evaluation metrics, enabling scalable causal inference for pricing, subsidies, and marketing.
Causal InferenceModel ServingSpark
0 likes · 23 min read

This article explains how Baidu processes internet‑scale content by applying deep AI‑driven understanding, detailing cost‑optimization, efficiency improvements, model‑service frameworks, resource‑scheduling systems, and batch‑compute platforms that together enable trillion‑level indexing and feature extraction.
AI EngineeringBatch ComputingHTAP storage
0 likes · 16 min read

This article details Xiaohongshu's end‑to‑end GPU‑centric transformation for search‑related machine‑learning models, covering model characteristics, training and inference frameworks, system‑level GPU and CPU optimizations, multi‑card and compilation techniques, and future directions for scaling large sparse and dense models.
GPU optimizationModel ServingXiaohongshu
0 likes · 16 min read

This guide walks you through creating a sidecarless AI demo on Alibaba Cloud Service Mesh ASM, covering environment setup, multi‑model serving with KServe, PVC storage, InferenceService configuration, business service deployment, gateway and waypoint creation, traffic routing rules, and OIDC single sign‑on integration.
AIASMKServe
0 likes · 28 min read

This article details Baidu Search's engineering practice for trillion‑scale content understanding, covering cost and efficiency challenges, model‑service framework, batch‑compute platform, resource‑scheduling system, HTAP storage design, and concrete optimization techniques such as multi‑process Python serving, dynamic batching, and two‑stage scheduling.
BaiduHTAPModel Serving
0 likes · 18 min read

This article examines the governance of WeChat's NLP algorithm microservices, outlining the management, performance, and scheduling challenges they face and presenting solutions such as automated CI/CD pipelines, dynamic scaling, DAG‑based service composition, a custom tracing system, the PyInter interpreter, and an improved load‑balancing algorithm.
CI/CDModel ServingNLP
0 likes · 12 min read

This guide explains how to deploy large language models on Alibaba Cloud's ACK using KServe for serverless inference, integrates Fluid for distributed data caching to cut cold‑start latency, provides step‑by‑step commands, performance benchmarks, and practical tips for production‑grade AI model serving.
FluidKServeKubernetes
0 likes · 22 min read

Huolala’s Big Data Intelligent Platform team describes how they built a cloud‑native, one‑stop AI solution on Kubernetes, integrating Flink‑based feature engineering, a multi‑tenant Zeppelin notebook, GPU‑aware training, and a unified model‑serving platform, while addressing resource isolation, storage persistence, and cross‑cloud deployment.
AI PlatformGPU schedulingKubernetes
0 likes · 17 min read

Xiaohongshu transformed its recommendation pipeline from daily to minute‑level updates by redesigning recall, ranking and feature‑joining components, deploying a base‑plus‑incremental training scheme, migrating Spark to Flink, rewriting services in C++, and optimizing RocksDB, which yielded over 10% longer dwell time, 15% more interactions and roughly 50% higher new‑note efficiency.
Model ServingReal-time TrainingRecommendation System
0 likes · 20 min read

The article outlines ten core components—data and feature pipelines, feature store, training and retraining pipelines, metadata store, serving infrastructure, production monitoring, reusable ML pipelines, workflow orchestration, CI/CT/CD, and end‑to‑end quality control—that together form a scalable, reliable architecture for modern machine‑learning systems.
MLOpsModel Servingfeature engineering
0 likes · 7 min read

This article describes the architecture, key features, dynamic deployment, performance optimizations, and real‑world results of a high‑throughput online inference platform that serves deep‑learning models for JD.com’s risk‑control decision engine, achieving near‑hundred‑fold latency improvements.
AIModel ServingRisk Control
0 likes · 11 min read

The article presents Snowball's model feature platform, detailing its motivation, architecture, feature lifecycle management, online engine design, optimization techniques, and the resulting improvements in feature iteration speed, reuse, and system stability for recommendation and search services.
Feature ManagementModel ServingRecommendation System
0 likes · 16 min read

This article reviews deep learning and AI frameworks, highlights challenges of online model serving, and presents Avior—a lightweight, distributed inference engine designed for high‑performance AI services, detailing its architecture, layer design, benchmark results, and future development plans.
AI FrameworksAviorModel Serving
0 likes · 8 min read

The article demonstrates how to integrate Paddle Serving’s new security‑gateway feature with the open‑source Kong API gateway and its Konga UI, using Docker‑Compose to create a secure, HTTPS‑encrypted, header‑authenticated AI model serving endpoint that hides internal services while supporting high‑concurrency inference.
AIAPI gatewayDocker
0 likes · 9 min read

This article details Weibo's large‑scale multimodal content understanding platform, covering its background, data and model heterogeneity challenges, the end‑to‑end workflow, GPU‑heterogeneous cluster design, resource scheduling, performance optimization for distributed training and online inference, and comprehensive monitoring to ensure stable, low‑latency AI services.
AI infrastructureGPU clusteringModel Serving
0 likes · 17 min read

This article details Weibo's multimodal content understanding platform, covering its massive data challenges, heterogeneous model support, standardized pipelines, platformization, workflow architecture, GPU heterogeneous cluster management, resource scheduling, performance optimization, and full‑stack monitoring to achieve stable, low‑latency AI services at scale.
GPU clusterModel ServingWeibo
0 likes · 18 min read

This article describes the challenges of deploying a large BERT model as an online service for 360 Search and details engineering optimizations—including framework selection, model quantization, knowledge distillation, stream scheduling, caching, and dynamic sequence handling—that dramatically improve latency, throughput, and resource utilization.
BERTFP16 quantizationGPU optimization
0 likes · 12 min read

The article describes JD's end‑to‑end online learning pipeline for retail search, covering the background, system architecture, real‑time feature collection, sample stitching, Flink‑based incremental training, parameter updates, and full‑link monitoring to achieve low‑latency, high‑accuracy model serving.
FlinkModel Servingfeature engineering
0 likes · 9 min read

This article examines the challenges of moving machine‑learning models from offline training to online serving, proposes a modular architecture—including model gateway, data source gateway, business service center, monitoring, and RPC components—to enable rapid model deployment, version management, traffic mirroring, gray‑release, and real‑time monitoring.
Model Servingmachine learningmonitoring
0 likes · 10 min read

This article presents a comprehensive technical solution for a distributed online inference system that packages machine‑learning models in Docker containers, orchestrates them with Kubernetes for fault‑tolerant, elastic scaling, and integrates model repositories, image registries, monitoring, and automated model selection to streamline deployment, updates, and resource management.
AIDockerKubernetes
0 likes · 15 min read

This article explains Alibaba’s User Interest Center approach to real‑time model serving, detailing how it separates offline sequence modeling from lightweight online inference, uses an online interest‑embedding store, and dramatically reduces latency for recommendation models such as DIEN and MIMN.
AlibabaEmbeddingModel Serving
0 likes · 8 min read

Meituan’s AI‑driven search combines the Augur online inference framework—offering stateless, distributed feature operators, transformers, and a DSL for rapid, high‑throughput model scoring—with the Poker platform for model training, versioning, and experimentation, together accelerating iteration, improving performance, and enabling advanced model‑as‑feature ensembles.
AI PlatformModel Servingfeature engineering
0 likes · 26 min read

Sunfish is Youzan’s integrated AI platform that unifies visual drag‑and‑drop model training, notebook‑based algorithm development, automated model management and one‑click publishing with a low‑latency, high‑availability “small‑box” inference service, enabling end‑to‑end deep‑learning workflows from data exploration to online recommendation and risk‑control deployment.
AI PlatformMLOpsModel Serving
0 likes · 17 min read


58 TechMar 27, 2020 · Artificial Intelligence
dl_inference is an open‑source inference platform that simplifies deployment of TensorFlow and PyTorch models in production, offering unified gRPC access, load‑balanced multi‑node serving, GPU/CPU options, customizable pre‑ and post‑processing, and extensible architecture for future AI workloads.
AI inferenceModel ServingPyTorch
0 likes · 11 min read

This article describes the end‑to‑end recommendation pipeline of the Province Money Fast Report app, covering business background, data collection, model training and evaluation, the evolution from FM to DeepFM, DIN, DCN, xDeepFM, ESMM and custom networks, as well as serving strategies and practical lessons learned.
CTR predictionModel ServingRecommendation System
0 likes · 28 min read

The article summarizes senior AI expert Wu Jianjun’s presentation on machine‑learning and deep‑learning engineering at Ping An Life, detailing the company’s big‑data platform, data processing pipelines, model training frameworks, distributed computing strategies, and production model‑serving architecture for financial applications.
Model Servingdeep learningdistributed computing
0 likes · 15 min read

The article introduces Ctrip’s Model Engine Platform, a comprehensive system that streamlines feature preparation, engineering, model management, and product orchestration to enable fast, reliable real‑time AI model serving across various business scenarios, while addressing common challenges such as manual data handling, offline‑only prediction, and long development cycles.
AI PlatformCtripFeature Management
0 likes · 15 min read
