Migrating Core Real-Time Indexing from Kafka to Apache Fluss: Xiaohongshu's Engineering Journey

1. Problem Background and Technical Evolution

Search, recommendation, and advertising systems are core online businesses for content platforms. They rely on real-time indexing pipelines to continuously distribute content features, user behaviors, and business wide tables generated by upstream computations to multiple downstream services, enabling new content and user interactions to quickly enter search indexes, recommendation features, or ad systems.

Xiaohongshu's public case uses major events like the World Cup as background to illustrate the pressure on real-time indexing pipelines: during high-traffic periods, search, feed, and live services must maintain data update capabilities without affecting online business due to index rebuilding or historical replay. The focus is on system load and service stability during peaks, not equating total event users with instantaneous concurrency.

In the original architecture, upstream continuously generated wide tables, and multiple downstream online services consumed required fields separately. For example, search services focused on content titles, tags, and text features; recommendation services on content attributes, user interactions, and behavioral features; ad services on another set of delivery and effect fields.

When multiple services repeatedly consume the same wide table but each uses only a subset of fields, three problems arise:

Read amplification. Consumers need different fields, but traditional Kafka consumer interfaces deliver data in complete message records. If a message payload is a wide-table row with dozens or hundreds of fields, even consumers using only a few fields must receive and parse the entire message.

Historical replay bypasses online message path. Index building consumes latest events and also replays historical data. If historical data resides in object storage but replay must go through Kafka brokers before reaching clients, it adds data read paths and broker load.

Historical scans compete with online consumption for resources. Batch index builds may continuously scan historical data at large scale, consuming network bandwidth and cache resources, thereby affecting latency-sensitive online consumers.

Xiaohongshu's official case notes that in their specific wide-table and downstream service combination, fan-out scale grew from 1x, 10x, 100x, reaching ~1000x in local workloads, making redundant transmission of irrelevant fields a significant cost to optimize. This fan-out multiplier describes the project workload, not a fixed Kafka performance parameter.

Therefore, the core of this migration is not simply judging "whether Kafka can still be used," but re-examining whether storage organization and data access patterns match.

The article unfolds in three layers: first explaining Kafka's cost sources under wide-table high-fan-out workloads; second analyzing Fluss's columnar stream storage, object storage historical reads, and Lakestream-Paimon collaboration; third dissecting migration methods like baseline stress testing, dual-run verification, gradual cutover, and platform capability building.

2. Core Concepts and Architecture Positioning

First, locate this real-time indexing pipeline, then clarify Kafka and Fluss technical boundaries.

3.1 Before Migration: Kafka Handles Real-Time Data Distribution

The original link can be abstracted as:

Upstream real-time computation → Wide table → Kafka → Search / Recommendation / Ad downstream online services

Meanwhile, batch index building and incremental index updates also need to read historical data. If historical data is stored in object storage but reads must go through Kafka brokers, historical scan traffic enters the online message distribution link.

Note a technical concept: Kafka is not a row-oriented storage engine in the database sense. Kafka's core abstraction is append-only logs and message records; brokers do not understand fields inside business messages. The so-called "row-oriented read amplification" more accurately occurs when messages are encoded as complete wide-table records and consumers read complete records via standard consumer interfaces, lacking native server-side field projection, resulting in redundant transmission and parsing of irrelevant fields.

If applications trim fields at the client side, they have already paid for message transmission and partial deserialization costs. Although message splitting, multi-topic design, or serialization format changes can reduce costs, these often require extra development and maintenance.

3.2 After Migration: Fluss Handles Columnar Stream Storage and Real-Time Data Access

Apache Fluss is a stream storage system for real-time analytics. It uses Apache Arrow IPC columnar format to organize default log data, allowing compatible consumers to request needed fields instead of always receiving entire wide-table records. Official documentation explicitly describes column projection's optimization effect on read I/O and data transfer overhead.

The post-migration main link abstracts to:

Upstream real-time computation → Apache Arrow write to Fluss → Downstream on-demand column projection reads → Search / Recommendation / Ad index services

Historical data access can use a different path:

Fluss remote log files → Historical index build tasks parallel read

Meanwhile, Lakestream continuously tiers data from Fluss into Apache Paimon via tiering mechanism, forming lake storage datasets for historical replay, offline analysis, and dual-run verification.

The architectural changes include three points:

Data organization changes. Fluss adopts columnar log format and supports read-side column projection.

Historical data access path changes. For eligible historical logs already stored in object storage, historical build tasks can read directly, reducing dependence on broker online transmission paths.

Real-time data and lake historical data collaborate. Fluss serves real-time access; Paimon as lake storage accepts tiered data. Both logically serve the same business dataset but remain different physical storage and access paths, not a single physical copy.

Thus, Fluss in this case is not a simple replacement of Kafka's transport interface, but leverages columnar storage, historical data reads, and lake storage collaboration to reorganize the core data pipeline's access paths.

4. Deep Dive into Underlying Principles

4.1 Why Does Wide-Table High Fan-Out Amplify Read Costs?

Assume a business wide table contains 100 fields, but a downstream service needs only 10 fields.

In typical Kafka consumption paths, if the entire wide-table row is encoded as one message, consumers usually must receive the complete record first, then extract needed fields. Ignoring extra compression, caching, or other optimizations, the consumer receives a message body containing large amounts of data it will not use.

If the same wide table is consumed independently by multiple services similarly, this invalid transmission repeats.

A simplified model illustrates:

This is an estimation model for understanding differences, ignoring compression, metadata, batch organization, and protocol overhead; it does not represent actual network traffic precisely equaling the formula.

Fluss's key advantage is supporting column projection in storage and read paths, so unneeded fields need not be read and transmitted in the same way. Official technical docs describe implementation based on Arrow columnar batches and server-side column pruning.

However, this mechanism cannot be interpreted as "larger fan-out guarantees multiplicative gains." Gains depend on multiple factors:

Actual pruned fields and their byte proportion in the wide table.

Number of consumers and their column access sets.

Data compression ratio and batch organization.

Client and server processing overhead.

Whether downstream reads actually execute via compatible column projection interfaces.

A more rigorous conclusion: When the same wide table is read by many consumers each needing few fields, native column projection usually has more opportunity to reduce overall network and compute costs. But if consumers almost all need full records, or the wide table is not actually wide, switching to columnar stream storage may yield limited benefits.

4.2 Why Should Historical Replay Not Long-Term Contend for Online Broker Resources?

Real-time indexing pipelines need not only to consume new events but also to read historical records when business rules change, indexes rebuild, or data repairs occur.

In the original architecture, if historical data resides in object storage but build tasks can only fetch data via Kafka brokers, an extra read path forms: Object storage → Broker → Index build client The problem is not just an extra network hop. Historical builds often feature large-scale, continuous reads and high concurrency, sharing broker network, disk, and cache resources with real-time message consumption. Under high load, online consumers may be affected.

Xiaohongshu's public case describes that after migration, index build tasks can directly read Fluss log files stored in object storage, reducing historical scan pressure on broker network and cache resources.

But direct object storage reads do not mean all historical data can naturally bypass Fluss services.

Implementation requires verification:

Whether corresponding historical logs have entered remote storage supporting direct reads.

Whether current Fluss version and deployment support that access path.

Whether index build side has correct object storage permissions, file parsing capability, and read progress management.

How to determine switch points between historical scans and continuous incremental consumption to avoid missing or duplicate processing.

Whether object storage throughput, request limits, and network capacity can support expected parallelism.

Also, direct object storage reads only isolate part of historical read traffic, not eliminating all resource contention. Object storage bandwidth, network egress, index build node CPU, and downstream index write capacity may still become bottlenecks.

Therefore, production environments should design and evaluate online incremental consumption and historical batch scanning as two distinct workloads , rather than relying on infinitely increasing broker capacity to carry all work.

4.3 Lakestream: How Fluss and Paimon Collaborate

Xiaohongshu's case also uses Apache Fluss's Lakestream architecture to continuously tier real-time data into Apache Paimon. Official articles note Paimon tables were used for data exploration and dual-run verification during migration; after production, lake data can further support historical replay, offline analysis, and batch index building.

We must avoid simplifying Lakestream into " the same data can be accessed in two systems at zero cost ".

Physically, Fluss and Paimon are different data storage systems. Data tiering means continuously writing qualifying data from Fluss into lake tables, typically forming physical data copies on different storage media. Their primary responsibilities differ:

Fluss: Real-time stream storage and supported low-latency access.

Paimon: Lake historical data, snapshot management, and stream-batch access.

Tiering Service: Continuously writes Fluss data into target lake format per tiering mechanism.

Index build tasks: Choose real-time consumption path or historical data read path as needed.

Fluss official docs state Tiering Service reads Fluss data incrementally and writes to lake storage, supporting mechanisms coordinated with lake-side commit progress. Specific semantics depend on deployment config and version used.

During migration, Paimon's value is especially clear: old and new index pipelines can process corresponding data separately, then compare output results using unified event identifiers, time windows, and business semantics.

But Paimon tables' existence alone cannot automatically prove both pipelines' results consistent. Verification must still handle event out-of-order, duplicate data, late data, field transformation differences, and data visibility latency.

Thus, Lakestream's main value is not eliminating all data copies, but letting real-time data enter a storage system with lake table semantics, giving historical replay, offline analysis, and migration verification a more unified data foundation.

5. Core Links and Key Processes

5.1 Before Migration: Online Consumption and Historical Scans Share Message Transport Link

Original link representation:

Upstream processing platform generates wide table → Kafka → Multiple online services

Simultaneously: Historical data → Kafka Broker → Index build tasks The first link handles real-time distribution; the second handles batch or incremental index building historical reads. When multiple online consumers frequently read full wide tables while historical tasks generate large scan traffic, they may compete for resources on the same transport layer.

Note that Kafka remains suitable for many event delivery, message buffering, and multi-consumer-group scenarios. The issue is not that it lacks reliable log capability, but that in this case the message access pattern does not fully match columnar real-time analytics and historical data read needs.

5.2 After Migration: Columnar Real-Time Consumption and Historical Data Reads Separated

New link abstraction:

This architecture optimizes three aspects separately:

Online consumers read data according to their field needs, reducing irrelevant field transmission and processing.

Eligible historical data can be read in parallel via remote storage access paths, avoiding all historical scan traffic going through online message brokers.

Fluss data can continuously tier to Paimon, so historical data use cases need not fully depend on real-time storage layer.

However, historical reads do not mean all index builds must go through Paimon. Specific choice depends on whether data resides in Fluss remote logs or lake tables, read patterns, query engine support, and performance requirements. The two paths can complement rather than replace each other.

5.3 Migration Process: Baseline Stress Testing, Dual-Run, Canary, and Gradual Cutover

Xiaohongshu's official article emphasizes the migration verification process: the team first conducted stress tests against key Kafka baselines, then performed extended dual-run verification, confirmed correctness before gradually onboarding core business traffic, while building platform capabilities like table management, multi-tenancy, monitoring alerts, data lineage, cluster operations, cross-cloud management, and backup.

For other teams, this process can be further refined into steps:

Phase 1: Establish baseline. Record old system throughput, P95/P99 latency, network traffic, CPU, consumer lag, failure recovery time, and index update latency under normal and peak loads.

Phase 2: Shadow dual-run. Let new system consume same data sources but not yet serve as unique production index writes. Compare results via independent downstream indexes or shadow namespaces.

Phase 3: Reconciliation verification. Align event primary keys, partitions, time windows, and deduplication rules; compare record missing, duplicates, field differences, and final index state.

Phase 4: Gradual cutover. Expand new system's coverage gradually by business domain, data table, or consumer group, ensuring each phase is independently observable and rollback-able.

Phase 5: Complete old link retirement. Only after stability, data correctness, performance, and recovery capabilities pass acceptance, gradually shut down old consumption paths.

The core of this method: migration is not a one-time message address swap, but making data correctness and online stability measurable, verifiable engineering goals.

6. Real Business Scenarios and Engineering Practice

6.1 Quantify Three Cost Categories Before Migration

Before evaluating Kafka-to-Fluss migration value, first quantify problem scale, not assume Fluss's columnar storage guarantees significant cost reduction.

First, quantify wide-table read amplification. Count each wide table's total fields, average message size, consumer count, each consumer's field read ratio, and network traffic. If multiple downstream services need only few fields, column projection potential gain is higher; if most consumers still need full records, gain may be smaller.

Second, quantify historical replay resource impact. Count batch index build and historical replay frequency, total data volume, concurrency, and online consumer latency changes during execution. Focus on identifying whether historical reads actually consume online broker network, cache, and service resources.

Third, quantify online vs historical load contention. Use peak-period broker network utilization, consumer latency, client lag, disk I/O, and historical task throughput to confirm if two workload types need separation. If bottlenecks are actually in downstream index writes or upstream wide-table compute resources, simply swapping stream storage may not solve the main problem.

6.2 Dual-Run Verification Needs Clear Reconciliation Criteria

The most important part of dual-run is not running two systems simultaneously, but proving they produce compliant results for identical business inputs.

Suggest covering at least these metrics:

Data completeness: Compare record counts, primary key sets, and partition coverage in same event window.

Data correctness: Compare key fields, type conversions, null handling, business rules, and output index content.

Duplicates and disorder: Verify new/old paths handle retries, failure recovery, duplicate events, and late data as expected.

Real-time performance: Compare end-to-end latency from upstream event generation to downstream index visibility.

Failure recovery: Actively simulate task restarts, network jitter, consumer lag, and historical replay interruptions; verify post-recovery output correctness.

Long-term stability: Dual-run should cover sufficient business cycles and representative peaks, not rely on a single short stress test.

For core businesses like search, recommendation, and ads, suggest using shadow indexes or independent test environments for dual-run verification to avoid old and new systems simultaneously producing duplicate updates to the same production index.

6.3 Platform Capabilities Must Mature Alongside Data Paths

Xiaohongshu's public case notes that beyond performance and correctness tests, capabilities like table management, multi-tenancy, monitoring alerts, data lineage, cluster operations, multi-cloud management, cross-region backup, and lake ingestion task management must be built synchronously.

Behind this is an easily underestimated issue: successful data path migration does not equal platform readiness for long-term operations.

For example, lacking multi-tenant resource governance may let a large-scale replay task affect other businesses; without data lineage and table management, change impact scope is hard to assess; missing backup and recovery drills may fail to meet production recovery targets during system failures.

Thus, migration acceptance should include:

Multi-tenant quotas and isolation.

Unified management of tables, consumers, and permissions.

Complete metrics, logs, tracing, and alerts.

Data lineage and upstream/downstream impact analysis.

Backup, disaster recovery, and DR drills.

Version upgrades, rollbacks, capacity scaling, and change management.

These may not directly reflect in throughput benchmarks but determine whether the system can run stably long-term.

6.4 From Single-Table Verification to Scaled Migration

Xiaohongshu's official article discloses that at publication time, single-table processing scale was ~1 billion records/day, ~10 TB; the team planned to continue gradual migration in H2 2026, targeting over 100 tables, infrastructure scale reaching hundreds of thousands of CPU cores.

Treat these two info types differently: the former is the project's disclosed single-table processing volume; the latter is a future plan described at publication, not confirmed as fully completed by this article alone.

What's more worth referencing is the migration rhythm:

First prove technical feasibility with key tables, then verify continuous writes, historical reads, long-running operation, and failure recovery, only then expand table count and production traffic scale.

Production environments should set explicit phase entry criteria, e.g.:

Single table runs stably and passes correctness checks.

Representative high-fan-out scenarios meet performance targets.

Historical replay no longer significantly impacts online SLA.

Failure restart and consumption recovery pass drills.

Operations, permissions, backup, and rollback processes are ready.

Only when these conditions are met should migration scope enter the next phase.

7. Performance, Cost, and Technology Selection

7.1 Project-Disclosed Performance Gains

Xiaohongshu's official blog published the following gains. They describe results under specific workload and config, not fixed Fluss metrics for all scenarios.

These numbers should not be directly used for capacity planning, nor can all gains be simply stacked. For example, 50% bandwidth reduction and 30% CPU reduction do not mean overall cost automatically drops 80%, because they may act on different resources and work paths.

A more reasonable approach is to build your own baseline model and conduct controlled tests on the same dataset, same business rules, same downstream concurrency, and similar hardware resources.

7.2 What Factors Determine Actual Gains?

Fluss's column projection reduces downstream reads of unneeded fields, but final effect depends on column pruning ratio and message encoding.

Direct object storage reads reduce historical scan dependence on brokers, but historical builds are still limited by object storage throughput, network bandwidth, parallelism, file layout, and downstream index write capacity.

Lakestream with Paimon can take on historical data and support offline uses, but also bring lake table maintenance, data tiering, and storage costs. Therefore, evaluating overall gains requires accounting for:

Real-time write and compute costs.

Network transfer and historical read costs.

Broker or stream storage cluster resource costs.

Paimon lake storage and data tiering costs.

Historical index build compute resources and runtime.

Platform transformation, migration verification, operations, and failure recovery costs.

Only by including these in a unified cost model can we judge if migration has overall economic value.

7.3 Technology Selection Conclusion

Selection conclusion: Fluss's value for this case comes from its match with wide tables, high fan-out, column projection, and historical read needs, not from being superior to Kafka in every dimension.

If existing Kafka architecture meets latency, throughput, and cost targets without obvious historical read contention, migration may not be justified. Conversely, if actual workloads are long constrained by full-message redundant transmission and historical replay competing with online services, evaluating columnar stream storage and more suitable historical read paths is necessary.

8. Production Issues and Troubleshooting Methods

Issue 1: New and Old Pipeline Data Cannot Align After Migration

Symptom: During dual-run, new and old pipeline output record counts, field content, or index results differ.

Possible causes: Different read start points, inconsistent consumption semantics, or differences in event deduplication, field projection, disorder handling, and late data strategies.

Troubleshooting steps:

Align source data range, event time windows, and consumption start positions.

Use business primary keys or stable event IDs to locate specific differences.

Check field types, nulls, serialization, schema changes, and business transformation logic.

Distinguish temporary differences due to data latency from permanent differences that never converge.

Fix principle: First find the earliest divergence point, then determine if it's read position, data processing rules, or downstream output issue. Cannot judge data fully consistent just because final counts match.

Prevention: Use unified reconciliation tools during dual-run covering duplicate events, failure recovery, late data, and representative peak cycles.

Issue 2: Historical Replay Still Slow After Direct Object Storage Reads

Symptom: Historical index builds have left main online broker path but task throughput still misses targets.

Possible causes: Insufficient object storage read capability, file organization unsuited for parallel access, column pruning not effective, or downstream index writes becoming new bottleneck.

Troubleshooting steps:

Confirm historical tasks actually use expected remote log read path.

Check if only necessary fields are read and relevant read interfaces support column projection.

Analyze object storage throughput, request latency, network utilization, and task parallelism.

Check index build side CPU, memory, batch writes, and backpressure.

If historical analysis needs lake-table-optimized read paths, evaluate reading from Paimon.

Fix principle: Adjust scan parallelism, column projection, file access, and downstream batch processing based on actual bottleneck. Cannot solve all read problems by simply increasing parallelism.

Issue 3: Online Bandwidth Does Not Achieve Expected Improvement

Symptom: After switching to Fluss, actual network bandwidth reduction far below expectations.

Possible causes: Consumers still need most fields, column projection not applied to actual read path, or network bottleneck lies in upstream writes, cross-datacenter transfer, or downstream processing.

Troubleshooting steps:

Count wide table field count, actual used fields, and field byte proportions.

Confirm read side truly enables column projection.

Compare network read/write volumes for same business traffic before/after switch.

Analyze producer, storage, and consumer resource consumption separately.

Fix principle: First verify if gain model holds, then decide whether to adjust field layout, consumer access patterns, or overall architecture. Cannot treat project's ~30%-90% bandwidth savings as achievable for every table.

Issue 4: System Migration Complete but Operational Capabilities Still Lacking

Symptom: Core jobs run on Fluss but table management, alerts, data lineage, failure recovery, or multi-tenant resource governance fall short of production requirements.

Possible causes: Team only migrated data read/write paths without synchronously building platform and operational capabilities.

Fix principle: Treat table management, monitoring, permissions, resource quotas, backup, and recovery as independent deliverables with clear owners, acceptance criteria, and drill results.

Prevention: Include platform maturity in migration acceptance, not supplement governance capabilities after cutting production traffic.

9. Advanced Interview and Architecture Thinking

Q1: Why may Kafka's network and compute costs be high in wide-table high-fan-out scenarios?

Answer: Kafka consumer interfaces work per message record. For full wide-table messages, if multiple consumers use only part of fields, they may repeatedly transmit and parse unneeded data. Fluss's columnar storage and column projection can reduce this overhead. Emphasize Kafka is not incapable of high fan-out loads, but such workloads may not fully leverage columnar access advantages.

Q2: How does Fluss reduce historical replay impact on online consumption?

Answer: In deployments supporting the capability, historical index build tasks can directly read Fluss log files stored in object storage, reducing historical scan dependence on broker online transmission paths. Gains depend on whether data resides in accessible remote storage, read path configuration, and object storage/downstream throughput capacity.

Q3: Why does Lakestream need to work with Paimon?

Answer: Fluss handles real-time stream storage; Paimon as lake table format accepts continuously tiered data, thereby supporting historical replay, offline analysis, and data exploration. During migration, lake tables can be used for dual-run result reconciliation, but must handle data freshness, record deduplication, and business semantic differences. Collaboration does not mean only one physical data copy exists.

Q4: What is the most critical engineering step when migrating core real-time pipelines?

Answer: First establish old system baseline, clarifying throughput, latency, cost, and stability metrics; second conduct new/old dual-run comparing data correctness; third gradually switch production traffic with rollback plan; finally perfect monitoring, table management, permissions, backup, and failure recovery capabilities.

Q5: Can CPU ~30% drop, bandwidth savings up to ~90%, peak throughput ~3x improvement be directly used for new system capacity planning?

Answer: No. These numbers are Xiaohongshu's public disclosures under specific workloads. Your gains depend on wide-table width, field usage ratio, fan-out scale, storage config, hardware/software environment, and downstream build efficiency. Verify via repeatable baseline stress tests and controlled experiments.

Q6: What common methodology can be extracted from the third round's eight industry cases?

These eight cases use different tech stacks but share a common architectural principle: first identify true structural costs, then make data storage and access patterns match business workloads, finally ensure production availability through data reconciliation, canary verification, and operational governance.

10. Summary, Knowledge Extraction, and Next Episode

This article, combined with Xiaohongshu's public sharing, analyzes key designs for gradual migration of core real-time indexing pipelines from Kafka to Apache Fluss.

Core knowledge extracts to five points:

Migration starts from access patterns, not component novelty. Wide-table high fan-out causing redundant field transmission, historical reads pressuring brokers, and real-time/lake historical data collaboration needs are the main drivers.

Columnar stream storage value lies in native column projection. Only when consumers indeed need only part of record fields does column pruning have chance to significantly reduce transmission and processing overhead.

Historical replay needs independent resource path evaluation. Direct object storage reads reduce online broker dependence but do not eliminate object storage, network, and downstream compute bottlenecks.

Lakestream is a real-time and lake data collaboration mechanism. Fluss and Paimon can serve same business dataset but must clarify physical storage, data tiering, and consistency boundaries.

Safe migration relies on engineering verification, not single performance tests. Baseline stress testing, long-cycle dual-run, correctness reconciliation, phased cutover, and rollback capability are necessary conditions for core online pipeline migration.

One-sentence conclusion: The key to migrating core real-time pipelines is not simply replacing Kafka with Fluss, but letting data organization, field access, and historical read paths match real business workloads, then using stress testing, dual-run, and gradual cutover to guarantee migration correctness.

This concludes the third round "Industry Deep Cases" episodes 17-24. Travel, finance, advertising, instant retail, connected vehicles, logistics, long-term rental SaaS, and cross-platform migration have presented core lakehouse problems from different angles: how storage and query match business, how real-time and historical collaborate, how tasks and data are governed, and how system changes land safely.

Round four will enter "Lakehouse + AI Frontier Topics", starting from Apache Fluss's real-time data foundation and Lakestream architecture, exploring relationships between real-time data, historical context, and AI applications, and further analyzing Paimon's role in AI data infrastructure. Future articles will combine Apache Fluss official release notes and public technical materials to distinguish released capabilities, architectural directions, and engineering practices yet to be verified.