Skip to content
FlowMQ Kafka: Decoupled Storage Cuts Costs 94% and Enables Seamless Scaling

FlowMQ Kafka: Decoupled Storage Cuts Costs 94% and Enables Seamless Scaling

FlowMQ Kafka separates compute and storage with S3 and RLog backends, delivering 94% cost savings on cloud object storage and eliminating manual rebalancing through segment-level replication and Flexible Quorum, outperforming Kafka and Redpanda in multi-AZ failure recovery.

java1234
java1234
java1234

Introduction

FlowMQ is a unified messaging system built from scratch in C++ that supports publish/subscribe, queue, and stream processing paradigms, with protocol compatibility for MQTT, Kafka, AMQP, and NATS. This article focuses on FlowMQ's Kafka protocol implementation, which not only maintains full compatibility with the standard Kafka protocol but also introduces a compute-storage separation architecture with two storage backends: S3 for low-cost cloud deployments and RLog for low-latency local deployments.

Problems with Apache Kafka

As data volumes and throughput grow, Apache Kafka faces four fundamental limitations:

Tight compute-storage coupling: Partitions and their replicas are bound to specific brokers, which handle both request processing and log storage, tying compute resources, disk capacity, and replica placement together.

Rising storage and network costs: Multi-replica reliability requires each replica to occupy independent block storage. In multi-AZ deployments, cross-AZ replication traffic adds significant network costs that scale with throughput and retention time.

Scaling requires data migration: Adding brokers does not automatically rebalance existing partition data. Partition reassignment moves replicas to new nodes, contending for disk and network bandwidth with live traffic; larger datasets mean longer migrations and greater business impact.

Capacity and load imbalance: Each replica is constrained by its broker's disk capacity. Large partitions cause uneven resource distribution, so individual nodes hit disk or network bottlenecks even when the cluster overall has free capacity.

Consequently, Kafka scaling involves not just adding resources but also replica redistribution, data migration, and ongoing operational overhead, increasing complexity and total cost as the business grows.

FlowMQ Kafka Architecture

FlowMQ Kafka adopts a compute-storage separation architecture. The Kafka protocol layer supports the standard Kafka protocol, ensuring compatibility with existing Kafka clients, topics, partitions, consumer groups, authentication, and retention policies, enabling seamless migration. The storage layer is backed by either S3 or RLog. Compute and storage layers can be scaled independently on demand.

Unified Kafka protocol layer with S3 and RLog dual storage backends
Unified Kafka protocol layer with S3 and RLog dual storage backends

FlowMQ Kafka-S3: Ultra-Low-Cost Cloud Deployment

The S3 backend leverages S3-compatible object storage for data, delivering extreme cost efficiency in the cloud:

Object storage offers a clear price advantage over self-managed multi-replica Kafka storage for the same data volume.

Object storage is billed by usage, whereas self-managed Kafka requires pre-provisioned high-spec storage capacity.

In multi-AZ deployments, FlowMQ reads and writes directly to object storage, eliminating expensive cross-AZ replication traffic.

For example, in an AWS multi-AZ deployment with 1 GB/s throughput, the monthly infrastructure cost for Apache Kafka is $286,636, while FlowMQ S3 costs only $18,138 — a 94% reduction. Users can compare costs for their target storage and throughput across cloud providers.

AWS 1 GB/s multi-AZ monthly infrastructure cost comparison
AWS 1 GB/s multi-AZ monthly infrastructure cost comparison

Moreover, the FlowMQ Kafka-S3 compute layer is completely stateless. Scaling in or out only requires adding or removing compute nodes, adjusting compute resources to business demand without pre-provisioning high-spec compute-storage nodes as in self-managed Kafka clusters.

FlowMQ Kafka-RLog: Low-Latency Local Storage

For on-premises deployments or scenarios demanding lower data access latency, FlowMQ provides the RLog storage backend. RLog is a custom distributed log engine designed around two characteristics of streaming data:

Data is always appended sequentially.

Historical data, once written, is never modified.

Accordingly, RLog divides a partition's log into multiple segments — zero or more sealed read-only segments and one active segment — and configures replication groups per segment . This allows historical segment replicas to be distributed across different nodes at a finer granularity, so capacity balancing no longer revolves around entire partition replicas.

Example of segment replica distribution for a single partition
Example of segment replica distribution for a single partition
In the diagram, S3 is the active segment. Each segment has three replicas: S1 on nodes 1,2,3; S2 on nodes 2,3,4; S3 on nodes 1,4,5.

This design yields several advantages over Apache Kafka:

Smaller replication-group granularity enables more even distribution across nodes.

Workloads are dynamically balanced in real time instead of being pinned to fixed nodes.

When a new node joins, new segments automatically expand onto it without manual reassignment or data copying.

On failure, the current active segment is immediately sealed and a new segment is created; the replication protocol is simplified, enabling faster failure recovery.

Flexible Quorum: Flexible Replica Configuration and Failure Recovery

RLog's replication protocol uses Flexible Quorum [1]. With total replicas N, write quorum W, and recovery quorum R, strong consistency is guaranteed when W + R > N, without requiring a majority.

In a two-AZ deployment scenario, the article compares Apache Kafka (ISR), Redpanda (Raft), and FlowMQ Kafka-RLog:

Apache Kafka (ISR): With four replicas (two per AZ), acks=all and min.insync.replicas=3, the cluster tolerates a single node failure. However, if an entire AZ fails, only two replicas remain, failing the write quorum. Recovery requires adding nodes and waiting for new replicas to catch up on historical logs, which can involve massive data copying and prolonged unavailability [2].

Redpanda (Raft): Raft requires a majority quorum for both writes and leader election. In a two-AZ setup with four replicas (two per AZ), the majority is three. Losing one AZ leaves only two replicas, unable to form a majority, rendering the cluster completely unavailable. Even adding standby nodes cannot bypass the original replication group's majority requirement [3]. Kafka's KRaft controller faces the same issue.

FlowMQ Kafka (RLog): Using a 4-3-2 configuration (N=4, W=3, R=2), after an AZ failure the two surviving replicas satisfy R=2, allowing reads and log recovery to proceed. Although the remaining two replicas cannot meet W=3 for writes, activating standby nodes or a standby AZ (automatable) immediately forms new replication groups for incoming data, restoring write availability without waiting for historical data migration. Write interruption is limited to failover and standby activation time, while strong consistency is maintained throughout.

Typical Use Cases

FlowMQ Kafka-S3: Massive Write and Long-Term Retention

Suitable for log retention, user behavior tracking, IoT device telemetry, audit trails, and monitoring metrics . These workloads feature continuous writes, large data volumes, and long retention periods, making them sensitive to per-unit storage cost, cross-AZ traffic cost, and elastic scaling capability. FlowMQ Kafka-S3 reduces long-term storage costs via object storage and supports independent scaling of compute nodes for peak/off-peak demand.

FlowMQ Kafka-RLog: Low Latency and High Availability

Suitable for financial trading, real-time risk control, real-time ordering, game messaging, industrial control, and private messaging platforms . These scenarios demand low read/write latency, high service availability, and fast failure recovery, often with on-premises deployment requirements. FlowMQ Kafka-RLog improves data distribution flexibility and failure recovery efficiency through fine-grained segment distribution and Flexible Quorum replication.

Conclusion

As business scale and data demands grow, messaging systems must better balance performance, cost, and scalability. FlowMQ Kafka's flexible architecture and storage options provide a new technical choice for Kafka workloads, offering greater flexibility for businesses of varying sizes and deployment requirements.

The actual data storage and scaling operations for both storage backends were demonstrated in a previous technical livestream. Readers can watch the replay for more technical details.

References

Heidi Howard, Dahlia Malkhi, Alexander Spiegelman. Flexible Paxos: Quorum Intersection Revisited . OPODIS 2016, LIPIcs 70, 25:1–25:14, 2017. DOI: 10.4230/LIPIcs.OPODIS.2016.25. (https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.OPODIS.2016.25)

Apache Kafka: Replication and Availability and Durability Guarantees (https://kafka.apache.org/40/design/design/#replication)

Redpanda: How Redpanda Works — Raft consensus algorithm (https://docs.redpanda.com/streaming/current/get-started/architecture/)

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Kafkamessaging systemcompute-storage separationS3RLogdistributed logFlexible QuorumFlowMQ
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.