Tagged articles
large-scale
62 articles · Page 1 of 1
The article details how a map‑intelligence platform transforms 180 days of nearly 10 trillion GPS points into an online service that returns the first matching trajectory in 0.35 seconds by combining ClickHouse, S2 spatial indexing, integer encoding, tiered storage and a streaming query pipeline.
ClickHouselarge-scales2
0 likes · 26 min read

This article explains Elasticsearch shard fundamentals, primary and replica roles, allocation rules, recovery and rebalance mechanisms, tuning parameters, best‑practice sizing, and presents real‑world production cases—including a 100,000‑shard cluster—along with concrete API commands for effective shard operations.
Cluster OperationsElasticsearchShard Management
0 likes · 28 min read

This article walks through the challenges of manual server deployment, explains why Ansible is ideal for large‑scale environments, and provides a complete reference architecture, optimized configuration, dynamic inventory scripts, modular playbooks, performance tuning, monitoring, security hardening, rollback mechanisms, cost analysis, and practical lessons learned for automating deployments across thousands of machines.
Ansibleautomationdeployment
0 likes · 20 min read

This article walks through the author's real‑world experience automating deployments across a thousand‑plus server cluster with Ansible, covering tool selection, architecture design, performance tuning, security practices, rollback mechanisms, cost‑benefit analysis, and common pitfalls, demonstrating how automation can boost efficiency tenfold.
Ansiblelarge-scaleperformance
0 likes · 18 min read

This article details Alibaba Cloud's intelligent operations team’s challenges and solutions for managing thousands of cloud‑native clusters, covering their multi‑layered operation architecture, GitOps workflow, infrastructure‑as‑code integration, and the role of AI‑driven intelligent operations in large‑scale environments.
GitOpsIaCKubernetes
0 likes · 23 min read

The paper introduces SOGAIC, a scalable overload‑aware graph‑based index construction system for billion‑scale vector similarity search that uses adaptive overlapping partitioning and load‑balanced distributed scheduling to cut construction time by 47.3% while maintaining high recall.
ANNDistributed SchedulingVector Search
0 likes · 13 min read

Alibaba Cloud, together with Zhejiang University, Alipay and Xieyun Technology, received the Zhejiang Province Science and Technology Progress First Prize for their cloud‑native large‑scale cloud‑edge collaborative platform, which addresses edge resource constraints, real‑time computing, and massive node management, and has been widely applied across multiple industries.
CNCFReal-time Computingcloud-native
0 likes · 5 min read

Alibaba Cloud, together with Zhejiang University and partners, earned the top Zhejiang Science & Technology Progress Award for its cloud‑native large‑scale cloud‑edge collaboration platform, which tackles edge device constraints, performance bottlenecks, and real‑time computing through asynchronous node/service management, dynamic load scheduling, and an open‑source edge platform, OpenYurt.
Cloud-Edge CollaborationEdge ComputingOpenYurt
0 likes · 5 min read

This guide outlines practical strategies for designing and optimizing network and storage in Kubernetes clusters of over 5,000 nodes, covering overlay networks, IP pool segmentation, bandwidth allocation, load balancing, security policies, distributed storage options, performance tuning, and reliable backup solutions.
IP PoolKubernetescloud native
0 likes · 9 min read

At QCon 2023, Baidu’s BES team detailed how their cloud‑native Elasticsearch service has been engineered for large‑scale vector search, describing architecture, C++ plugin integration, memory‑saving storage tricks, HNSW/IVF optimizations, filter strategies, and real‑world multimodal video and LLM knowledge‑base deployments.
AIBESElasticsearch
0 likes · 16 min read

This article introduces the distributed transactional KV store TiKV, explains its role as TiDB’s storage engine, details its multi‑layered architecture and Raft‑based consistency model, and discusses the performance and resource challenges encountered at massive data scales along with the engineering solutions implemented to address them.
Distributed DatabaseRaftTiKV
0 likes · 14 min read


AntTechOct 30, 2023 · Artificial Intelligence
AntM2C is a publicly released, billion‑sample click‑through‑rate (CTR) dataset covering five distinct Alipay business scenarios, providing both ID and rich multi‑modal (text and image) features to enable comprehensive evaluation of multi‑scenario, cold‑start, and multi‑modal CTR models at industrial scale.
CTRMulti-modallarge-scale
0 likes · 14 min read

This article details how JDV, JD.com’s internal visual dashboard platform, tackled the massive data‑intensive 618 promotion by implementing real‑time updates, cross‑midnight count stops, request‑state control, heartbeat monitoring, proxy data sources, and a suite of developer tools to ensure stability, performance, and rapid feature delivery.
Real-timedata platformlarge-scale
0 likes · 18 min read

This article presents a comprehensive overview of Baidu Intelligent Cloud's tape‑library solution, detailing tape and tape‑library fundamentals, the Aries cloud storage stack, data and access models, the end‑to‑end data flow, key architectural design choices, implementation details, and a real‑world case study demonstrating large‑scale cold‑data storage, backup, and retrieval performance.
AriesCold DataData Archiving
0 likes · 28 min read

The forum gathers senior experts from Baidu, Meituan, Ant Group, Xiaomi and Baidu Cloud to share practical experiences and future trends on constructing ultra‑large‑scale file, block, KV and NoSQL storage systems, focusing on low‑cost, high‑performance solutions and architectural challenges.
KV storageStorageblock storage
0 likes · 8 min read

In October 2022 Tencent finished migrating its flagship services—including QQ, WeChat, and Honor of Kings—to a cloud‑native architecture spanning over 50 million CPU cores, overcoming millisecond‑level upgrade, stateful in‑place refresh, massive cross‑region scaling, and heterogeneous hardware by deploying the TKEx platform’s sidecar upgrades, three‑container patterns, Global Scaler Operator, machine‑type abstraction, and Clusternet‑based application‑centric orchestration, boosting CPU utilization to 65 % and establishing China’s largest cloud‑native practice.
Container UpgradeTencentcloud-native
0 likes · 19 min read

Google’s monolithic repository, managed by the proprietary Piper system and accessed via the cloud‑based CitC client, stores over a billion files and billions of lines of code, supports tens of thousands of engineers, and relies on trunk‑based development, extensive tooling, and strict security to enable large‑scale, efficient software development.
DevOpsGooglePiper
0 likes · 17 min read

Alibaba’s advertising pipeline progressed from low‑dimensional quantization partitions to hierarchical tree indexes, then to graph‑based HNSW structures—including multi‑category, multi‑level graphs and a BlazeOp‑driven scoring service—dramatically boosting recall efficiency, scalability and maintainability while meeting strict latency constraints.
HNSWRecallVector Search
0 likes · 13 min read

The Karmada community's Cloud Native Days China 2022 session in Nanjing gathered over 30 enterprises and developers to share multi‑cloud, multi‑cluster production practices, large‑scale testing results, and real‑world implementations from Huawei Cloud, vivo, Hurricane Engine, China Mobile, DaoCloud, and Zhejiang University, highlighting Karmada's scalability and ecosystem growth.
KarmadaKubernetesMulti-Cluster
0 likes · 9 min read

BattleBit Remastered is a low‑poly, large‑scale multiplayer FPS that supports up to 254 players per match, offering extensive class and weapon options, a minimalist UI, destructible environments, and strong team‑communication tools, all while running on minimal hardware requirements.
BattleBit RemasteredUIfps
0 likes · 12 min read

This article describes how to efficiently test billions of API parameter combinations by replacing naive nested loops with a queue‑based concurrent approach, dynamically controlling QPS, and addressing memory‑pressure issues using thread‑safe data structures.
API TestingJavalarge-scale
0 likes · 8 min read

This article details Meituan's real‑world challenges with a 15,000‑node Kafka deployment and explains the application‑layer and system‑layer optimizations—such as disk balancing, migration pipeline acceleration, fetcher isolation, RAID acceleration, cgroup isolation, and an SSD‑based cache—that together dramatically cut read/write latency and simplify large‑scale cluster management.
Meituancluster managementlarge-scale
0 likes · 23 min read


ITPUBSep 24, 2022 · Operations
This article explains how Alibaba Cloud Log Service implements a billion‑scale task scheduling framework for its observability platform, covering background, task types, design goals, architecture, key design points, and practical examples such as aggregation jobs and various scheduling scenarios.
Log ServiceTask Schedulinglarge-scale
0 likes · 22 min read

At the 16th OSDI conference, Tencent’s WeChat team presented the award‑winning Ekko system—a groundbreaking, ultra‑low‑latency model‑update solution for massive recommendation workloads that dramatically speeds up updates, supports over a trillion‑scale models, and has already boosted user engagement across billions of daily users.
WeChatlarge-scalelow-latency
0 likes · 5 min read

This talk presents recent advances in multi-step reasoning over large-scale, noisy knowledge graphs, introducing the Query2Box model that uses box embeddings for complex queries and the SMORE framework that enables efficient multi-hop inference on massive graphs through scalable query generation, embedding computation, and training pipelines.
AIMulti-hop ReasoningQuery2Box
0 likes · 14 min read

ByteDance’s cloud‑native ecosystem combines a multi‑layered architecture, dynamic resource over‑provisioning control, hybrid online‑offline scheduling, and federated cluster management to boost container utilization from 23% to 63%, reduce costs by 40%, and support massive events like the 2021 Spring Festival Gala.
Hybrid Deploymentcloud nativelarge-scale
0 likes · 16 min read

Vivo’s rapid migration to Kubernetes across multiple data centers required a secure, efficient, and reliable way to manage thousands of nodes, leading them to develop a custom k8s‑operator that streamlines cluster deployment, CI testing, declarative APIs, and automated repair for large‑scale cloud‑native environments.
Cluster AutomationDevOpsKubernetes
0 likes · 3 min read

This talk explains Ant Group’s serverless‑based front‑end platform, detailing the current front‑end architecture, challenges of large‑scale financial services, and three core efficiency ideas—document‑as‑code, function‑level deployment, and integrated plugin capabilities—to streamline development and ensure safe production.
BFFdocument-as-codefrontend
0 likes · 9 min read

This article examines the practical aspects of open‑sourcing Twitter’s recommendation algorithm, covering the platform’s data model, timeline views, ranking features, a TypeScript pseudocode illustration, and the major engineering challenges of scale, real‑time processing, reliability, and security.
AlgorithmRecommendation SystemTwitter
0 likes · 14 min read

This article explains how Alibaba Cloud's self‑built KubeProbe combines universal link probing and targeted inspections to detect, diagnose, and remediate issues in massive multi‑cluster Kubernetes environments, improving reliability and reducing on‑call overhead.
ChatOpsInfrastructureKubernetes
0 likes · 19 min read

The paper presents a guarantee‑based, distributed allocation framework for Alibaba’s off‑site brand contract ads that extends the SHALE algorithm with effect‑driven objectives and explicit over‑allocation constraints, solves dual variables via coordinate descent, and employs adaptive probability‑based pacing to meet volume guarantees while significantly boosting average CTR.
Online Optimizationallocationguaranteed display
0 likes · 11 min read

This article presents a comprehensive overview of building and applying product knowledge graphs for e‑commerce, covering background, recent advances in graph neural network‑based entity alignment, online prediction pipelines, data construction, evaluation metrics, attribute extraction, and future research directions.
Graph Neural Networkattribute extractione-commerce
0 likes · 23 min read

The article introduces the Dual Vector Foil (DVF) algorithm system, which decouples index construction from model training to enable lightweight, high‑precision large‑scale recall using arbitrary complex models, and details its two‑stage and one‑stage solutions, graph‑based retrieval implementation, performance optimizations, and experimental results.
Algorithmdeep learninggraph indexing
0 likes · 28 min read

This article examines Ant Group's strategies for achieving high availability of the etcd key‑value store in a massive 10,000‑node Kubernetes cluster, detailing challenges, performance metrics, filesystem upgrades, tuning parameters, operational platform insights, and future directions for distributed etcd deployments.
Kubernetesetcdlarge-scale
0 likes · 21 min read

This article describes how Du Xiaoman Financial designed, built, and continuously optimized a massive graph platform—including data governance, graph learning, query performance, data import, and online deployment—to improve credit risk assessment using billions of nodes and edges, and shares practical lessons on graph databases, distributed training, and real‑time inference.
DGLFinancial AnalyticsGraph
0 likes · 19 min read

This article presents an overview of graph neural networks, explains their computational framework, analyzes space and time complexities, and proposes ten practical strategies—including edge avoidance, dimensionality reduction, selective iteration, memory baking, distillation, partitioning, sparse computation, routing, and cross-sample feature sharing—to significantly lower the cost of large‑scale GNN processing.
computational complexitydeep learninglarge-scale
0 likes · 14 min read

JD Logistics architect Chen Haolong detailed the design, scalability strategies, and operational practices behind the billion‑level asynchronous messaging system that powered JD.com’s massive 11.11 shopping festival, revealing how the platform handled unprecedented traffic and ensured reliability.
JD Logisticsasync messaginge‑commerce
0 likes · 2 min read

This article details Alibaba's practical experience deploying Service Mesh at massive scale, covering architectural evolution, key challenges, traffic interception, hot‑upgrade mechanisms, performance optimizations, and operational tooling that together enable reliable, low‑overhead service communication in a cloud‑native environment.
EnvoyIstiocloud native
0 likes · 22 min read

Alibaba’s large‑scale Service Mesh deployment faces challenges such as smooth technology evolution, business‑technical balance, technical debt, massive sidecar operations, and scaling, which it addresses through staged architecture evolution, traffic‑transparent interception, hot upgrades, and open‑source contributions to Istio and Envoy.
EnvoyIstiocloud native
0 likes · 19 min read

This article shares Weibo’s experience in building and evolving its recommendation algorithms, covering the recommendation scenario, machine learning workflow, feature engineering, model upgrades, large‑scale challenges, deployment via the Weiflow platform, and the capabilities of its machine‑learning infrastructure.
Weibofeature engineeringlarge-scale
0 likes · 14 min read

This article surveys how major Chinese tech firms such as JD.com, Ctrip, Didi, and 58.com deploy and evolve Elasticsearch clusters to handle massive order data, log analysis, real‑time monitoring, and security tasks, detailing architecture choices, shard strategies, multi‑cluster designs, and performance optimizations.
ElasticsearchSearch Architecturebig data
0 likes · 11 min read

This article presents JTM, a joint optimization framework that simultaneously learns a tree‑based index and a deep scoring model to overcome the limitations of traditional recommendation pipelines, demonstrating significant recall improvements on Amazon Books and Alibaba UserBehavior datasets through hierarchical user interest modeling and efficient tree learning.
deep learningjoint optimizationlarge-scale
0 likes · 19 min read

Ant Financial’s article details how its large‑scale Kubernetes management system—built on a meta‑cluster, end‑state operators, and a Kube‑on‑Kube design—ensures reliable creation, upgrade, and self‑healing of thousands of nodes, while providing gray‑scale changes, risk assessment, and fault‑tolerant automation.
KubernetesOperatorSelf-Healing
0 likes · 15 min read

This article explains how Ant Financial designs and operates a large‑scale, highly available Kubernetes management platform—detailing its architecture, core operators, desired‑state controllers, fault‑self‑healing mechanisms, risk mitigation, and practical Q&A for production environments.
KubernetesSelf-Healingautomation
0 likes · 16 min read

This article explains how Ant Financial designs and operates a large‑scale, highly available Kubernetes management platform—using end‑state driven operators, custom CRDs, self‑healing mechanisms, and risk‑mitigation strategies—to reliably run thousands of nodes and dozens of business clusters in production.
Kube-on-KubeKubernetesSelf-Healing
0 likes · 15 min read

This article details Meitu's large‑scale personalized recommendation pipeline, describing the business scenario, challenges of massive data, latency and long‑tail distribution, and the application of deep learning techniques such as Item2vec, YouTubeNet, dual‑tower DNN, NFM, NFwFM and multi‑task learning to improve click‑through rate, conversion and user engagement.
Personalizationdeep learninglarge-scale
0 likes · 20 min read

This article presents a comprehensive overview of the Tree‑based Deep Match (TDM) algorithm, describing the evolution of retrieval technology, the limitations of traditional Match‑Rank pipelines, the design of a one‑stage tree‑indexed deep matching model, its training methodology, performance gains on public datasets, and its deployment in Alibaba’s advertising and e‑commerce platforms.
TDMlarge-scalerecommendation systems
0 likes · 23 min read

This article explains how Ant Financial designs a highly reliable, end‑state‑driven Kubernetes management platform that handles lifecycle operations, node self‑healing, and risk‑controlled changes for clusters with tens of thousands of nodes, using operators, custom resources, and a meta‑cluster architecture.
KubernetesSelf-Healingcluster management
0 likes · 9 min read

WeChat, China’s top social app with over a billion users, has applied DevOps practices to dramatically improve development efficiency, code quality, and accelerate the feedback cycle from requirements to delivery, while confronting real‑world challenges in tooling, processes, reliability, and automation costs.
Continuous DeliveryWeChatlarge-scale
0 likes · 3 min read

Ant Financial tackles the challenge of managing dozens of Kubernetes clusters and over a hundred thousand worker nodes by employing a meta‑cluster with Kube‑on‑Kube and Node Operators, enabling automated lifecycle management, scaling, upgrades, and fault recovery for both master components and worker nodes.
KubernetesOperatorautomation
0 likes · 12 min read

The article shares practical experience on operating massive Kubernetes clusters, focusing on three stability questions, data collection and visualization, and a suite of operational tools to achieve reliable, high‑availability services in production environments.
Cluster OperationsKuberneteslarge-scale
0 likes · 12 min read

DeepHash is Meitu’s large‑scale short‑video analysis and retrieval system that converts deep‑learned visual features into compact binary hash codes via a MobileNet‑based CNN and triplet‑loss training, enabling fast, robust similarity search across billions of videos with sub‑second latency and minimal storage.
feature extractionlarge-scalemultimedia retrieval
0 likes · 15 min read

This comprehensive guide explains what large‑scale website operations entail, outlines the product lifecycle involvement of ops engineers, details the technical and personal skills required, and discusses current challenges, future prospects, and key technologies such as cluster management, monitoring, fault handling, and automation.
DevOpsInfrastructureLinux
0 likes · 18 min read


58 TechJun 1, 2018 · Backend Development
This article explains how 58.com’s in‑house C++ search kernel ESearch was architected to provide second‑level real‑time indexing, high‑concurrency low‑latency querying, flexible ranking models, and efficient storage structures for billions of daily queries across massive classified data.
BackendC++large-scale
0 likes · 13 min read

This article details the architecture and practical techniques behind WeChat's large‑scale monitoring system, covering lightweight data collection, classification of real‑time, non‑real‑time and user‑specific metrics, anomaly detection algorithms, automated configuration, and high‑performance storage solutions for billions of events per minute.
Real-timeStoragedata collection
0 likes · 14 min read

This article explains how Alibaba’s Sunfire monitoring platform processes terabytes of logs per minute, uses a pull‑based architecture with Brain‑Reduce‑Map roles, tackles scalability and reliability challenges, and outlines future directions such as MQL standardization and intelligent baselines.
Real-timelarge-scalelog processing
0 likes · 17 min read


ITPUBNov 14, 2017 · Operations
Alibaba’s Dragonfly (蜻蜓) is a self‑developed P2P file distribution platform that dramatically speeds up massive file and container image delivery, reduces bandwidth consumption, supports intelligent compression and flow control, and has become a core infrastructure component powering billions of transactions during major events like Double 11.
Container ImagesFile DistributionInfrastructure
0 likes · 20 min read

Alibaba’s Dragonfly (蜻蜓) leverages P2P networking, intelligent compression, and flow control to dramatically accelerate large‑scale file and container image distribution, reducing bandwidth usage by over 99%, achieving up to 57× speedup, and supporting tens of thousands of concurrent hosts during peak events like Double 11.
Container ImagesFile DistributionP2P
0 likes · 19 min read

This article shares insights from Tencent’s SNG operations leader on building effective continuous feedback loops for massive‑scale services, covering monitoring, alerting, operational metrics, multi‑dimensional analysis, and practical DevOps techniques to improve reliability, availability, and automated self‑healing.
Continuous FeedbackDevOpslarge-scale
0 likes · 26 min read

This article explains how Tencent built a billion‑scale, generalized recommendation system by designing a reusable algorithm library, deploying a low‑latency, highly available real‑time streaming platform (R2), and offering a cloud‑based recommendation engine that simplifies integration for internet businesses.
AIReal-time ComputingRecommendation System
0 likes · 11 min read

This article explains how 360's Qcmd, a Golang‑based real‑time command execution system, overcomes SaltStack's limitations to reliably manage tens of thousands of servers with high success rates, flexible scripting, detailed monitoring, and efficient message handling.
Command ExecutionQcmdautomation
0 likes · 7 min read

The article analyzes Facebook's massive image storage challenges and explains the Haystack architecture, detailing its components—Directory, Store, and Cache—how it reduces I/O, manages metadata, and handles read/write operations at billions‑scale while also addressing CDN dependency and fault tolerance.
BLOB storageBackendFacebook
0 likes · 10 min read
