Tagged articles
real-time indexing
15 articles · Page 1 of 1
This article details Xiaohongshu's migration of its core real-time indexing pipeline from Kafka to Apache Fluss, covering architectural drivers like read amplification and historical replay contention, Fluss's columnar storage and Lakestream integration with Paimon, and a phased migration approach including baseline testing, shadow dual-run, and gradual cutover.
Apache FlussApache PaimonKafka
0 likes · 50 min read

This article explains why traditional databases like MySQL struggle with massive data, introduces Elasticsearch’s advantages, and details a practical architecture using Hive, Canal, and Otter to achieve near real‑time indexing of petabyte‑scale datasets with minimal latency.
CanalData Transfer ServiceElasticsearch
0 likes · 20 min read

This article explains why traditional databases like MySQL struggle with massive datasets, introduces ElasticSearch’s inverted‑index architecture, and details a practical pipeline using Hive, wide tables, binlog, Canal, and Otter to achieve near real‑time indexing for petabyte‑level data.
CanalHiveOtter
0 likes · 19 min read

Learn how to construct near‑real‑time Elasticsearch indexes for petabyte‑scale datasets by comparing MySQL limitations, leveraging inverted indexes, using Hive and wide tables, and employing binlog‑based pipelines with Canal and Otter to achieve second‑level index updates.
CanalElasticsearchHive
0 likes · 18 min read


JavaEdgeMar 6, 2024 · Backend Development
This article details the design of a low‑latency video recommendation system, covering requirements, overall architecture, data pipelines, consistency handling, write smoothing, and performance optimizations such as multi‑level caching and Elasticsearch tuning.
ElasticsearchFlinkcaching
0 likes · 13 min read

This article explains how to construct near real-time Elasticsearch indexes for petabyte‑scale datasets by comparing MySQL limitations, introducing ES fundamentals, leveraging Hive and wide tables, and employing binlog‑based tools like Canal and Otter for low‑latency data synchronization.
CanalElasticsearchHive
0 likes · 22 min read

The article details Xianyu’s end‑to‑end product‑search architecture—covering tokenization, indexing, online request flow, offline index building, multi‑datacenter active‑active deployment, and supporting ad and debugging systems—and explains how expanding searcher capacity, separating query engines, grading updates, and diffusing auxiliary‑table writes together reduced latency from hours to near‑zero, enabling real‑time search.
e-commercereal-time indexingscalability
0 likes · 11 min read

This article explains why traditional databases like MySQL struggle with petabyte‑scale queries, introduces Elasticsearch’s architecture and inverted‑index advantages, and details a practical pipeline using Hive, Canal, Otter, and a custom DTS service to achieve near‑real‑time indexing for massive data volumes.
CanalElasticsearchMySQL
0 likes · 19 min read

This article explains how to construct near real‑time Elasticsearch indexes for petabyte‑level datasets by comparing MySQL limitations, describing Elasticsearch fundamentals, and detailing a pipeline that uses Hive, wide tables, MySQL binlog, Canal, and Otter to achieve second‑level index updates.
CanalElasticsearchOtter
0 likes · 18 min read

The article describes a real‑time video recommendation indexing system that replaces 30‑minute batch builds with an Elasticsearch‑based service, integrates prior and posterior data pipelines, ensures consistency via locking and version checks, enables zero‑downtime upgrades, smooths write spikes, and boosts recall performance through multi‑level caching and ES tuning, delivering sub‑40 ms latency and significant business growth.
ElasticsearchFlinkcaching
0 likes · 13 min read


ITPUBJul 25, 2020 · Backend Development
SimSvr is a high‑performance, distributed feature‑retrieval component designed for recommendation systems that supports billion‑scale indexes, sub‑millisecond query latency, real‑time and batch updates, multi‑model AB‑testing, and advanced filtering, all while running on Tencent's production workloads.
ANNfeature retrievalreal-time indexing
0 likes · 17 min read

This article explains how to build a real-time search engine indexing pipeline using Flink, covering background, batch versus incremental indexing strategies, a hybrid architecture that merges both approaches, and a concrete cloud‑based implementation involving MySQL binlog, Logtail, SLS, and Elasticsearch.
ElasticsearchFlinkbig data
0 likes · 5 min read


58 TechJan 25, 2019 · Backend Development
The article summarizes the evolution and redesign of Zhihu's search engine, details 58 Group's high‑performance uesearch architecture, real‑time indexing mechanisms, cloud‑native deployment with Kubernetes, and highlights key technical insights and future directions for large‑scale search systems.
Distributed SystemsKubernetesRust
0 likes · 9 min read

This article describes how Ctrip's hotel R&D team tackled growing order‑volume challenges by sharding the database, building a real‑time Elasticsearch index, implementing a custom replication pipeline, and applying various write‑ and read‑optimizations to achieve low latency and stable performance.
ElasticsearchPerformance OptimizationSQL Server
0 likes · 10 min read


58 TechJun 1, 2018 · Backend Development
This article explains how 58.com’s in‑house C++ search kernel ESearch was architected to provide second‑level real‑time indexing, high‑concurrency low‑latency querying, flexible ranking models, and efficient storage structures for billions of daily queries across massive classified data.
BackendC++large-scale
0 likes · 13 min read
