Tagged articles
large-scale data
29 articles · Page 1 of 1
The article explains how to solve the classic massive‑data deduplication problem of 4 billion QQ numbers within a 1 GB memory limit by analyzing memory requirements, comparing BitMap and Bloom filter approaches, and detailing a step‑by‑step BitMap implementation with its pros and cons.
AlgorithmBitmapInterview
0 likes · 5 min read

The article introduces AISHPerf, the first open‑source benchmark for AI‑infra operations agents built on nearly a hundred‑billion real‑world ops records, detailing its data pipeline, multi‑layer coverage, evaluation metrics, experimental results that show current models lag behind human experts, and future plans to expand and refine the benchmark.
AI OpsEvaluation MetricsGPU cluster
0 likes · 16 min read

Holi‑Spatial introduces a fully automatic pipeline that transforms raw video streams into high‑quality 3D geometry, depth, masks, 3D boxes, instance descriptions, grounding and spatial QA, producing the 4‑million‑item Holi‑Spatial‑4M dataset and substantially improving VLM spatial reasoning performance.
3D reconstructionICML 2026Multimodal Dataset
0 likes · 14 min read

NeoteAI, a Fudan‑affiliated startup, raised nearly 100 million yuan to advance its visual‑tactile sensor, large‑scale data platform, and VTLA model that together give robots precise touch perception, boosting fine‑grained manipulation success rates above 90% in industrial settings.
AI ModelEmbodied AIRobotics
0 likes · 10 min read

Paxini, together with JD Cloud, Tencent Cloud, and Baidu Intelligent Cloud, launches the world’s first hundred‑billion‑scale, full‑modal, high‑degree‑of‑freedom embodied AI data cloud mall, offering instant online data procurement, end‑to‑end model training pipelines, and validated performance gains in both lab and real‑world robot tasks.
Embodied AIModel TrainingMultimodal Data
0 likes · 13 min read

Tencent's LFM4Ads introduces a full‑representation, multi‑granular knowledge transfer framework that moves user, item, and cross representations from a large foundation model to downstream tasks, achieving up to 2.45% platform GMV uplift across more than ten advertising scenarios.
Representation Learningads recommendationfoundation model
0 likes · 12 min read

This article describes the challenges and technical solutions for synchronizing billions of order records from a relational database to Elasticsearch, including multi‑source data reading, dynamic rate limiting, retry strategies, SPI‑based service integration, environment isolation, health‑checking, smooth migration, and structured logging, all implemented in a backend service called ECP.
JavaSPIbackend service
0 likes · 21 min read

This article explains how to use a Bloom filter backed by Redis bitmap and Roaring Bitmap sharding to efficiently filter 800 million OpenID queries, covering memory planning, hash function selection, code implementation, and performance‑tuned batch write strategies.
BitmapBloom FilterRoaring Bitmap
0 likes · 13 min read


ITPUBOct 1, 2023 · Backend Development
This article explains how to design a scalable architecture for classification‑info platforms that handle billions of rows, ten‑thousand attributes, and hundred‑thousand QPS by using vertical partitioning, unified post, category, and search services, along with compressed JSON extensions and external indexing.
Vertical Partitioninglarge-scale datascalable architecture
0 likes · 12 min read

This article explains the background, key scenarios, functional components, third‑party payment capabilities, implementation logic, rule‑engine usage, and large‑scale data handling strategies of a checkout system, providing a comprehensive view of its backend architecture and operational considerations.
Backendcheckoutlarge-scale data
0 likes · 14 min read

This article presents a comprehensive overview of multimodal pre‑training, describing its motivation, architecture choices, large‑scale Chinese image‑text dataset construction, training optimizations, performance benchmarks, downstream applications, and a Q&A session that highlights practical deployment considerations.
Computer VisionPretrainingdeep learning
0 likes · 16 min read


AntTechNov 28, 2022 · Information Security
This article details the evolution, architecture, and key technologies of Ant Group's anti‑intrusion platform, explaining how it handles trillion‑level data streams for intrusion detection, performs multi‑dimensional risk assessment and attribution, and enables rapid, automated security incident response across massive enterprise environments.
Security Automationanti-intrusioninformation security
0 likes · 15 min read

Angel Graph is a high‑performance, fault‑tolerant distributed graph computing framework developed by Tencent, featuring scalable node‑metric, community‑detection, and graph‑neural‑network algorithms optimized for billion‑node, trillion‑edge datasets, and demonstrated through practical applications in intelligent financial risk control.
Community DetectionRisk Controldistributed systems
0 likes · 20 min read

Facing a challenge of managing roughly one million server-side images and 180 client images, the TOOSIMPLE team built a high-performance backend using fingerprinting, parallel processing, mmap-SSE2 acceleration, and sparsemap indexing, achieving sub-second response times while ensuring correct ordered display.
golanghashinglarge-scale data
0 likes · 12 min read

This article demonstrates two approaches for efficiently importing massive CSV data into MySQL using Python: a direct pymysql method with chunked inserts and a concise pandas‑SQLAlchemy method, comparing performance, code complexity, and offering tips for further speed improvements.
Data IngestionMySQLPandas
0 likes · 5 min read


ITPUBJun 9, 2022 · Artificial Intelligence
This article details the design, data pipeline, model architecture, loss functions, and evaluation metrics of a large‑scale multi‑label image classification system built for 58.com, showing how it improves semantic similarity detection, recommendation, and content moderation across diverse business domains.
Computer Visionasymmetric lossdeep learning
0 likes · 18 min read

This article examines a business requirement to filter up to 100,000 items from a pool of tens of millions, presenting and evaluating four technical solutions—multithreaded ClickHouse pagination, Elasticsearch scroll‑scan, an ES‑HBase hybrid, and RediSearch + RedisJSON—along with performance data and implementation details.
HBaseRediSearchRedisJSON
0 likes · 10 min read

The article introduces WuDaoMM and WuDaoCorpora 2.0, massive Chinese‑English multimodal datasets—including 650 million image‑text pairs, 3 TB of text, 93 TB of images, and 181 GB of dialogue—detailing their composition, formats, access options, and potential research applications.
Chinese AIMultimodal DatasetWuDaoMM
0 likes · 6 min read

This article presents Meituan's large‑scale Kafka deployment, describing the current state and challenges of massive data ingestion, detailing latency‑reduction techniques, cluster‑level optimizations, SSD‑based caching, isolation strategies, full‑link monitoring, lifecycle management, and future directions for high availability.
KafkaMeituanRead‑Write Latency
0 likes · 22 min read

This article explains four practical methods—sorting, hash map, file splitting, and bitmap—to deduplicate 4 billion QQ numbers within a 1 GB memory limit, and provides extended exercises on sorting, finding the median, top‑K, and duplicate detection for massive datasets.
AlgorithmBitmapInterview
0 likes · 8 min read

Zhihu’s Moneta service stores over a trillion rows and faces massive write and read loads; this article explains why TiDB was chosen, how its architecture and features such as HTAP, Raft, Titan and table partitioning enable millisecond‑level query latency, high availability, and seamless scaling.
Distributed DatabaseHTAPTiDB
0 likes · 15 min read


21CTOMay 18, 2021 · Big Data
This article explains Baidu's multimodal retrieval system, detailing the offline and online pipelines, the image processing and indexing platform (Imazon), its architecture, key technologies such as ANN and GPU models, and the optimization practices that enable massive daily image ingestion and real‑time search at billion‑scale.
BaiduMultimodal SearchPipeline Optimization
0 likes · 13 min read

This article details Baidu's large‑scale image processing and multimodal retrieval system, describing its offline‑online architecture, massive data ingestion pipeline, ANN search techniques, performance metrics, infrastructure components, and a series of optimizations for throughput, cost, and reliability in a high‑volume streaming environment.
BaiduImazonMultimodal Retrieval
0 likes · 12 min read

This article details how Zhihu tackled the massive data and latency challenges of its Moneta service by migrating from MySQL sharding and MHA to the distributed NewSQL database TiDB, describing the new three‑tier architecture, performance gains, migration tactics, and expectations for TiDB 3.0.
HTAPNewSQLTiDB
0 likes · 13 min read

This article describes how Zhihu's Moneta service, which stores over a trillion rows of user‑read data, migrated from MySQL sharding to the distributed NewSQL database TiDB to achieve high availability, horizontal scalability, millisecond‑level query latency, and improved overall system performance.
HTAPTiDBdatabase scalability
0 likes · 13 min read

Zhihu’s Moneta service, handling over 1.3 trillion rows and billions of daily writes, migrated from MySQL sharding to TiDB, achieving millisecond query latency, high availability, and horizontal scalability, while sharing architectural choices, performance metrics, migration challenges, and future expectations for TiDB 3.0.
HTAPMySQL MigrationTiDB
0 likes · 16 min read

This article presents an in‑depth technical overview of ByteGraph, ByteDance’s self‑built distributed graph database and its accompanying graph‑computing engine, covering graph data characteristics, the directed‑property graph model, API design, three‑tier system architecture, storage strategies using KV stores and B‑Trees, hotspot handling, indexing, and future research directions.
B+TreeByteGraphGremlin
0 likes · 33 min read

Alibaba, together with top Chinese universities and research institutes, unveiled the Cangjingge Knowledge Engine project, detailing its massive data assets, five‑module architecture, large‑scale knowledge construction techniques, and initial deployments in safety and tourism knowledge graphs to boost AI applications.
AIAlibabaKnowledge Engine
0 likes · 9 min read


21CTOMar 22, 2017 · Artificial Intelligence
The Youku Tudou data team overhauled its video recommendation system by moving ranking from offline to online, detailing architectural changes, advantages, challenges, feature handling, offline evaluation, and model weight fusion to improve scalability and user experience.
AB TestingAIRecommendation System
0 likes · 7 min read
