Large Scale | BestHub

Xiaohongshu Tech REDtech

May 22, 2025 · Artificial Intelligence

Scalable Overload-Aware Graph-Based Index Construction for 10‑Billion‑Scale Vector Similarity Search (SOGAIC)

The paper introduces SOGAIC, a scalable overload‑aware graph‑based index construction system for billion‑scale vector similarity search that uses adaptive overlapping partitioning and load‑balanced distributed scheduling to cut construction time by 47.3% while maintaining high recall.

ANNLarge ScaleVector Search

0 likes · 13 min read

Scalable Overload-Aware Graph-Based Index Construction for 10‑Billion‑Scale Vector Similarity Search (SOGAIC)

Alibaba Cloud Infrastructure

Nov 22, 2024 · Cloud Native

Large‑Scale Cloud‑Edge Collaborative Technology Based on Cloud‑Native Wins Zhejiang Province Science and Technology Progress Award

Alibaba Cloud, together with Zhejiang University, Alipay and Xieyun Technology, received the Zhejiang Province Science and Technology Progress First Prize for their cloud‑native large‑scale cloud‑edge collaborative platform, which addresses edge resource constraints, real‑time computing, and massive node management, and has been widely applied across multiple industries.

CNCFCloud NativeContainer

0 likes · 5 min read

Large‑Scale Cloud‑Edge Collaborative Technology Based on Cloud‑Native Wins Zhejiang Province Science and Technology Progress Award

Baidu Geek Talk

Nov 8, 2023 · Databases

BES Engineering Practices for Large‑Scale Vector Database Scenarios

At QCon 2023, Baidu’s BES team detailed how their cloud‑native Elasticsearch service has been engineered for large‑scale vector search, describing architecture, C++ plugin integration, memory‑saving storage tricks, HNSW/IVF optimizations, filter strategies, and real‑world multimodal video and LLM knowledge‑base deployments.

AIBESElasticsearch

0 likes · 16 min read

BES Engineering Practices for Large‑Scale Vector Database Scenarios

DataFunSummit

Nov 2, 2023 · Databases

Understanding TiKV: Features, Architecture, and Large‑Scale Operational Challenges

This article introduces the distributed transactional KV store TiKV, explains its role as TiDB’s storage engine, details its multi‑layered architecture and Raft‑based consistency model, and discusses the performance and resource challenges encountered at massive data scales along with the engineering solutions implemented to address them.

Distributed DatabaseLarge ScalePerformance Optimization

0 likes · 14 min read

Understanding TiKV: Features, Architecture, and Large‑Scale Operational Challenges

AntTech

Oct 30, 2023 · Artificial Intelligence

AntM2C: A Large-Scale Multi‑Scenario Multi‑Modal CTR Prediction Dataset from Alipay

AntM2C is a publicly released, billion‑sample click‑through‑rate (CTR) dataset covering five distinct Alipay business scenarios, providing both ID and rich multi‑modal (text and image) features to enable comprehensive evaluation of multi‑scenario, cold‑start, and multi‑modal CTR models at industrial scale.

Large Scalectrdataset

0 likes · 14 min read

AntM2C: A Large-Scale Multi‑Scenario Multi‑Modal CTR Prediction Dataset from Alipay

JD Tech

Oct 10, 2023 · Operations

Technical Case Study of JDV Visual Dashboard Platform for the 618 Promotion

This article details how JDV, JD.com’s internal visual dashboard platform, tackled the massive data‑intensive 618 promotion by implementing real‑time updates, cross‑midnight count stops, request‑state control, heartbeat monitoring, proxy data sources, and a suite of developer tools to ensure stability, performance, and rapid feature delivery.

Large Scaledata-platformmonitoring

0 likes · 18 min read

Technical Case Study of JDV Visual Dashboard Platform for the 618 Promotion

DataFunTalk

Sep 22, 2023 · Big Data

Design and Practice of Baidu's Tape Library Storage Architecture Based on the Aries Cloud Storage System

This article presents a comprehensive overview of Baidu Intelligent Cloud's tape‑library solution, detailing tape and tape‑library fundamentals, the Aries cloud storage stack, data and access models, the end‑to‑end data flow, key architectural design choices, implementation details, and a real‑world case study demonstrating large‑scale cold‑data storage, backup, and retrieval performance.

AriesCold DataDistributed Storage

0 likes · 28 min read

Design and Practice of Baidu's Tape Library Storage Architecture Based on the Aries Cloud Storage System

DataFunSummit

Jun 21, 2023 · Databases

Forum on Building Ultra‑Scale Storage Systems: Insights from Baidu, Meituan, Ant Group, Xiaomi and Baidu Cloud

The forum gathers senior experts from Baidu, Meituan, Ant Group, Xiaomi and Baidu Cloud to share practical experiences and future trends on constructing ultra‑large‑scale file, block, KV and NoSQL storage systems, focusing on low‑cost, high‑performance solutions and architectural challenges.

Distributed SystemsKV storageLarge Scale

0 likes · 8 min read

Forum on Building Ultra‑Scale Storage Systems: Insights from Baidu, Meituan, Ant Group, Xiaomi and Baidu Cloud

Tencent Cloud Developer

May 10, 2023 · Cloud Native

Tencent's Large‑Scale Cloud‑Native Migration: Challenges and Solutions

In October 2022 Tencent finished migrating its flagship services—including QQ, WeChat, and Honor of Kings—to a cloud‑native architecture spanning over 50 million CPU cores, overcoming millisecond‑level upgrade, stateful in‑place refresh, massive cross‑region scaling, and heterogeneous hardware by deploying the TKEx platform’s sidecar upgrades, three‑container patterns, Global Scaler Operator, machine‑type abstraction, and Clusternet‑based application‑centric orchestration, boosting CPU utilization to 65 % and establishing China’s largest cloud‑native practice.

Cloud NativeContainer UpgradeKubernetes

0 likes · 19 min read

Tencent's Large‑Scale Cloud‑Native Migration: Challenges and Solutions

Continuous Delivery 2.0

May 8, 2023 · Operations

Google’s Monolithic Code Repository: Scale, Architecture, and Practices

Google’s monolithic repository, managed by the proprietary Piper system and accessed via the cloud‑based CitC client, stores over a billion files and billions of lines of code, supports tens of thousands of engineers, and relies on trunk‑based development, extensive tooling, and strict security to enable large‑scale, efficient software development.

DevOpsGoogleLarge Scale

0 likes · 17 min read

Google’s Monolithic Code Repository: Scale, Architecture, and Practices

Alimama Tech

Feb 8, 2023 · Artificial Intelligence

Evolution of Recall Indexes in Alibaba Advertising: From Quantization to Graph-based HNSW

Alibaba’s advertising pipeline progressed from low‑dimensional quantization partitions to hierarchical tree indexes, then to graph‑based HNSW structures—including multi‑category, multi‑level graphs and a BlazeOp‑driven scoring service—dramatically boosting recall efficiency, scalability and maintainability while meeting strict latency constraints.

HNSWIndexingLarge Scale

0 likes · 13 min read

Evolution of Recall Indexes in Alibaba Advertising: From Quantization to Graph-based HNSW

NetEase LeiHuo Testing Center

Dec 9, 2022 · Game Development

BattleBit Remastered – An In‑Depth Analysis of Its Large‑Scale Multiplayer FPS Design

BattleBit Remastered is a low‑poly, large‑scale multiplayer FPS that supports up to 254 players per match, offering extensive class and weapon options, a minimalist UI, destructible environments, and strong team‑communication tools, all while running on minimal hardware requirements.

BattleBit RemasteredFPSLarge Scale

0 likes · 12 min read

BattleBit Remastered – An In‑Depth Analysis of Its Large‑Scale Multiplayer FPS Design

FunTester

Oct 24, 2022 · Backend Development

Optimizing Large-Scale API Parameter Combination Testing with Concurrency and QPS Control

This article describes how to efficiently test billions of API parameter combinations by replacing naive nested loops with a queue‑based concurrent approach, dynamically controlling QPS, and addressing memory‑pressure issues using thread‑safe data structures.

API testingConcurrencyJava

0 likes · 8 min read

Optimizing Large-Scale API Parameter Combination Testing with Concurrency and QPS Control

DataFunSummit

Jul 26, 2022 · Artificial Intelligence

Multi-step Reasoning over Large-scale Knowledge Graphs: Query2Box and SMORE Framework

This talk presents recent advances in multi-step reasoning over large-scale, noisy knowledge graphs, introducing the Query2Box model that uses box embeddings for complex queries and the SMORE framework that enables efficient multi-hop inference on massive graphs through scalable query generation, embedding computation, and training pipelines.

AILarge ScaleQuery2Box

0 likes · 14 min read

Multi-step Reasoning over Large-scale Knowledge Graphs: Query2Box and SMORE Framework

Efficient Ops

Jun 23, 2022 · Cloud Native

How Vivo Scales Kubernetes: Automated Multi‑Cluster Management with a Custom Operator

Vivo’s rapid migration to Kubernetes across multiple data centers required a secure, efficient, and reliable way to manage thousands of nodes, leading them to develop a custom k8s‑operator that streamlines cluster deployment, CI testing, declarative APIs, and automated repair for large‑scale cloud‑native environments.

Cloud NativeCluster AutomationDevOps

0 likes · 3 min read

How Vivo Scales Kubernetes: Automated Multi‑Cluster Management with a Custom Operator

Laravel Tech Community

Jun 6, 2022 · Artificial Intelligence

What an Open‑Source Twitter Algorithm Would Look Like: Architecture, Data Model, and Engineering Challenges

This article examines the practical aspects of open‑sourcing Twitter’s recommendation algorithm, covering the platform’s data model, timeline views, ranking features, a TypeScript pseudocode illustration, and the major engineering challenges of scale, real‑time processing, reliability, and security.

Large ScaleTwitteralgorithm

0 likes · 14 min read

What an Open‑Source Twitter Algorithm Would Look Like: Architecture, Data Model, and Engineering Challenges

Alimama Tech

Feb 9, 2022 · Artificial Intelligence

Online Allocation Strategies for Guaranteed Display Advertising: Modeling, Distributed Solving, and Adaptive Pacing

The paper presents a guarantee‑based, distributed allocation framework for Alibaba’s off‑site brand contract ads that extends the SHALE algorithm with effect‑driven objectives and explicit over‑allocation constraints, solves dual variables via coordinate descent, and employs adaptive probability‑based pacing to meet volume guarantees while significantly boosting average CTR.

Large Scaleadvertisingallocation

0 likes · 11 min read

Online Allocation Strategies for Guaranteed Display Advertising: Modeling, Distributed Solving, and Adaptive Pacing

DataFunTalk

Dec 29, 2021 · Artificial Intelligence

Entity Alignment in Product Knowledge Graphs: Techniques and Applications

This article presents a comprehensive overview of building and applying product knowledge graphs for e‑commerce, covering background, recent advances in graph neural network‑based entity alignment, online prediction pipelines, data construction, evaluation metrics, attribute extraction, and future research directions.

Large Scaleattribute extractione-commerce

0 likes · 23 min read

Entity Alignment in Product Knowledge Graphs: Techniques and Applications

DataFunTalk

Dec 13, 2021 · Artificial Intelligence

Dual Vector Foil (DVF): Decoupled Index and Model for Large‑Scale Retrieval

The article introduces the Dual Vector Foil (DVF) algorithm system, which decouples index construction from model training to enable lightweight, high‑precision large‑scale recall using arbitrary complex models, and details its two‑stage and one‑stage solutions, graph‑based retrieval implementation, performance optimizations, and experimental results.

Large ScaleRecommendation systemsalgorithm

0 likes · 28 min read

Dual Vector Foil (DVF): Decoupled Index and Model for Large‑Scale Retrieval

Efficient Ops

Nov 9, 2021 · Operations

How Ant Group Scales etcd for 10k‑Node Kubernetes Clusters: High‑Availability Secrets

This article examines Ant Group's strategies for achieving high availability of the etcd key‑value store in a massive 10,000‑node Kubernetes cluster, detailing challenges, performance metrics, filesystem upgrades, tuning parameters, operational platform insights, and future directions for distributed etcd deployments.

High AvailabilityKubernetesLarge Scale

0 likes · 21 min read

How Ant Group Scales etcd for 10k‑Node Kubernetes Clusters: High‑Availability Secrets