This article describes how to design and implement a gray (canary) messaging capability for RabbitMQ and Kafka, covering background, gray scenarios, two consumption strategies, and detailed production and consumption flows with code snippets for header tagging, requeue handling, and consumer group management.
This guide explains how Know Streaming adds a non‑native Kafka feature that lets users batch‑scale replicas for one or multiple topics, customize target brokers, preview and edit the reassignment plan, and throttle the operation to minimize impact on the cluster.
This guide walks through preparing the environment, installing and configuring Elasticsearch, Logstash, Kibana, Kafka, and Filebeat on a CentOS 7 system, including necessary code snippets, user permissions, and troubleshooting tips for a fully functional logging and data pipeline.
This article explains Kafka's distributed publish‑subscribe architecture, detailing its core components, underlying mechanisms with Zookeeper coordination, key features such as high throughput and fault tolerance, and common application scenarios like log collection, user activity tracking, and stream processing.
This article introduces Elasticsearch, Logstash, Kibana, and Kafka, explains their roles in data collection, analysis, and visualization, and provides a complete Docker‑Compose configuration to deploy these components together for scalable log processing and search solutions.
This article details a lightweight intelligent monitoring platform built on the open‑source WVP framework, describing its modular architecture, edge‑computing workflow with KubeEdge, SIP registration process, real‑time streaming setup, core features, and technical innovations such as MongoDB adoption and flexible pod scheduling.
This article introduces Debezium, an open‑source low‑latency change data capture platform that streams database row changes via Kafka, explains its architecture and common scenarios such as cache invalidation and CQRS, and provides step‑by‑step Docker commands to install ZooKeeper, Kafka, MySQL and the Debezium connector.
When starting a new Kafka consumer group, an unexpected SyncGroup error occurs due to a RecordTooLargeException, and the article walks through log inspection, identifies the oversized __consumer_offsets record, and resolves the issue by increasing the message.max.bytes configuration.
This article compares Elasticsearch and ClickHouse for log analytics, presents cost‑benefit calculations, and provides a step‑by‑step deployment guide for Zookeeper, Kafka, Filebeat, and ClickHouse to build a scalable, low‑cost data analysis platform for SaaS services.
This article describes the motivation, requirements, design, and implementation of a lightweight asynchronous message processing framework built by ByteDance to handle near‑real‑time metadata changes for DataLeap's Data Catalog, detailing its architecture, thread model, state management, delay handling, monitoring, and operational experiences.
This article describes how Meituan’s data platform tackles the growing challenges of a 15,000‑plus‑node Kafka deployment by detailing current bottlenecks, latency‑reduction techniques across application and system layers, large‑scale cluster management strategies, and future directions for robustness and cloud‑native migration.
This article walks through the complete Kafka consumer JoinGroupRequest lifecycle, detailing how the client builds and sends the request, how the group coordinator selects the coordinator node, processes unknown and known members, elects a leader, chooses a partition assignment protocol, and transitions group states.
This article outlines the design and implementation of a near‑real‑time metric management platform at Liulishuo, detailing its data flow—from Kafka ingestion through Flink‑SQL processing into Hudi tables, Trino querying, metric configuration, lineage, visualization, alerting, scheduling, and future optimization plans.
Interviewers frequently probe candidates on ensuring 100% message reliability in MQ systems like Kafka or RabbitMQ, and this guide walks through the underlying concepts, potential loss points, detection mechanisms, idempotent design, handling backlog, and practical ID generation strategies to ace such questions.
The article walks through the Kafka consumer heartbeat lifecycle, detailing how the HeartbeatThread is started, paused, and used to send heartbeat and LeaveGroup requests, how the GroupCoordinator validates and processes these requests, the client’s response handling, and the resulting state transitions illustrated with diagrams.
The article describes how Wix engineers built a robust, Kafka‑based event‑driven messaging infrastructure for over 1,400 microservices, detailing patterns such as consumption and projection, end‑to‑end streaming with websockets, in‑memory KV stores, schedule‑and‑forget jobs, exactly‑once transactions, and event aggregation to achieve scalability, resilience, and low‑latency data access.
This article recounts the end‑to‑end design, implementation, and iterative optimization of a billion‑scale full‑link tracing system at Ping An Health using SkyWalking, covering why full‑link monitoring is needed, the selection of SkyWalking, architecture choices, performance bottlenecks, and the roadmap for future enhancements.
This article details Meituan's Kafka deployment, describing the current massive scale and associated challenges, and presents a series of optimizations—including read/write latency reductions, application‑ and system‑level improvements, large‑scale cluster management strategies, full‑link monitoring, service lifecycle management, and future directions—to enhance performance, reliability, and scalability of the streaming platform.
This article explains how Wix uses Kafka‑based event‑driven messaging to decouple microservices, improve scalability, and achieve exactly‑once processing through patterns such as consume‑and‑project, end‑to‑end event streams, in‑memory KV stores, scheduled jobs, transactional events, and event aggregation.
Shopee’s off‑platform ads delay service combines Redis Zsets for expiration tracking, HBase for payload storage, and Kafka for queuing to reliably process up to 6 million tasks per minute with minute‑level delays ranging from one minute to thirty days, achieving horizontal scalability, fault tolerance, and a 75 % reduction in Kubernetes resource usage.
The article explains the role of Kafka's consumer group coordinator and consumer coordinator, details how group coordinators are selected, and walks through the JoinGroup, SyncGroup, LeaveGroup, and heartbeat processes, as well as partition assignment strategies and common Q&A.
Kafka’s architecture consists of brokers forming clusters, producers publishing to topics split into partitions with replicas, consumers organized in groups pulling messages by offset, ZooKeeper managing metadata, and log‑based storage using append‑only files, indexes, and zero‑copy, while configurable acknowledgment, batching, and replication ensure high throughput and fault‑tolerant reliability.
The article details a practical, reusable enterprise architecture for data classification and grading that combines scanning tools, a rule‑engine with hot‑updates, a high‑performance identification service, and a security enforcement platform, addressing massive real‑time data volumes, diverse storage types, cross‑department isolation, and compliance with China’s data security laws.
This tutorial walks you through the fundamentals of Zookeeper and Kafka, then provides detailed, code‑rich instructions for deploying both services on a Kubernetes cluster using Helm, configuring persistence, exposing ports, monitoring with Prometheus and Grafana, and finally cleaning up the installations.
This article details how Meituan's Kafka platform, serving over 15,000 machines and handling petabytes of daily traffic, faces read/write latency, slow nodes, and large‑scale cluster management challenges, and describes a series of application‑layer, system‑layer, and operational optimizations—including disk balancing, migration pipelines, fetcher isolation, consumer async, SSD caching, isolation strategies, full‑link monitoring, lifecycle management, and TOR disaster recovery—to improve performance and reliability.
This article describes how Meituan's massive Kafka deployment—over 15,000 machines and petabytes of daily traffic—faces scalability challenges such as slow nodes, load imbalance, and resource contention, and details the multi‑layer optimizations applied at the application, system, and cluster‑management levels to reduce read/write latency and improve reliability.
This guide details a step‑by‑step migration from a cloud‑hosted Kafka service to a self‑managed on‑premises cluster, covering requirements, a data‑synchronization strategy using Kafka MirrorMaker, consumer offset alignment, potential pitfalls, and practical scripts to ensure a smooth transition.
This article introduces Apache Kafka and Apache Pulsar, compares their core features such as publish/subscribe messaging, storage, real‑time pipelines, and stream processing, outlines key characteristics like high throughput, scalability and fault tolerance, and explains fundamental concepts and architecture components unique to each platform.
This article explains why introducing a message queue like Kafka helps decouple systems and control traffic, then dives into the three key questions of detecting, locating, and preventing message loss, offering concrete monitoring methods, configuration settings, and troubleshooting steps for producers, brokers, and consumers.
This article demonstrates how to build a lightweight, plug‑and‑play flow‑orchestration system for microservices by installing Nacos with Docker, configuring SpringBoot services with Kafka and Nacos, and using dynamic Nacos listeners to adjust Kafka topics at runtime without redeployment.
Lancer, Bilibili’s real‑time streaming backbone, has evolved from a monolithic Flume pipeline to a log‑id‑isolated, Kubernetes‑native architecture where Go edge agents feed synchronous Kafka‑proxied gateways into per‑logid topics processed by dedicated Flink‑SQL jobs, delivering exactly‑once, back‑pressured, highly scalable data ingestion for billions of daily requests.
The article explains how Redis Streams offers a lightweight, memory‑based alternative to Kafka, detailing its features, consumer‑group model, performance advantages, and suitable use cases while acknowledging scenarios where a full‑featured message queue remains preferable.
This article explains how to automatically generate large volumes of realistic test data for statistics, billing, and IoT scenarios by leveraging API calls, Metersphere timers, CSV-driven scripts, and custom Kafka producers, reducing manual effort while preserving data accuracy.
This article explains Kafka's message delivery semantics, analyzes potential data‑loss scenarios across producer, broker, and consumer components, and provides concrete configuration and coding practices—such as idempotent producers, proper ACK settings, replication factors, and manual offset commits—to maximize message durability and reliability.
This article dissects why a well‑tuned Kafka cluster can process up to 20 million messages per second, examining producer batching and custom protocols, broker page‑cache, file layout and zero‑copy techniques, as well as consumer group strategies that together unlock its high throughput.
This article explains how HuoLala’s real‑time R&D platform redesigns Flink metric collection, routing, and alerting using a custom Kafka‑based pipeline, flexible dashboards, and multi‑level metric governance to improve observability, reduce latency, and ensure data quality.
This article explains what message queues are, outlines key application scenarios such as asynchronous processing, message distribution, and traffic shaping, compares popular solutions like ActiveMQ, RabbitMQ, RocketMQ, and Kafka across performance and reliability dimensions, and provides practical guidance for selecting the most suitable queue for different business needs.
This article provides a comprehensive overview of Apache Kafka, explaining its distributed message‑queue architecture, the role of topics and partitions, producer and consumer workflows, leader election, offset management, consumer‑group rebalancing, delivery semantics, transaction processing, file organization, and key configuration settings.
This article explains the four key techniques—page‑cache usage, sequential disk writes, zero‑copy transfers, and partitioned segment indexing—that enable Kafka to achieve exceptionally high write performance, detailing how each mechanism reduces latency and maximizes throughput.
This article compares Elasticsearch and ClickHouse for log analytics, presents cost analysis, and provides detailed deployment instructions for Zookeeper, Kafka, Filebeat, and ClickHouse to build a private, high‑performance backend data platform for SaaS services.
This article explains how transactions, consistency, and consensus interrelate in distributed systems, reviews replication models, details ACID properties, internal consistency models, consensus algorithms, and Kafka’s configuration and transaction support, and highlights verification tools like TLA+ and Jepsen for ensuring correctness.
The article surveys three data‑replication models—master‑slave, multi‑master, and leaderless—explains how they enable scalability and fault‑tolerance, and examines core distributed‑system challenges such as partial node failures, unreliable networks, and unsynchronized clocks, while stressing safety‑liveness trade‑offs and design techniques like quorum and timeouts.
This article details Huolala's challenges and solutions in governing hybrid‑cloud databases, covering background, pain points, platform architecture, MySQL/Redis/Kafka implementations, automation, cost optimization, and the evolving role of DBAs in the cloud era.
Meituan’s data platform uses Kafka as a unified cache and distribution layer, and as the cluster grew beyond 15,000 brokers and daily traffic exceeded 30 PB, the team tackled slow nodes, read/write latency, large‑scale cluster management, full‑link monitoring, service lifecycle, and future cloud‑native deployment challenges.
This article explores the fundamentals and practical benefits of buffering in Java applications, covering file I/O streams, Logback asynchronous logging, and Kafka producer batching, while providing code examples, configuration tips, optimization strategies, and cautions to avoid data loss.
This article thoroughly explains Kafka's network layer by dissecting the complete send and receive workflow, covering message pre‑sending, selector polling, write and read operations, buffer management, and callback handling, all illustrated with core source code snippets.
This article explains how Flink's Broadcast State feature can be used to dynamically update processing rules and enrich streaming events with user information from MySQL, showing configuration, code examples, key considerations, and runtime results that demonstrate real‑time adaptability without restarting the job.
This guide compares three log‑collection methods—SLS, NAS, and Kafka—available in Alibaba Cloud Serverless App Engine, explaining their architectures, ideal use cases, configuration details, and how to avoid common pitfalls such as log loss and rotation issues.
The article explains fundamental concepts, common use cases, and selection criteria for four popular message‑queue platforms—Kafka, RabbitMQ, Pulsar, and RocketMQ—detailing each system’s architecture, strengths, and drawbacks so readers can match workload, scalability, ordering, latency, and operational needs to the most suitable middleware.
This comprehensive guide explains Kafka’s origins, core architecture, data structures, write and read workflows, operational commands for topic and consumer‑group management, and practical performance‑tuning tips such as disk layout, JVM settings, flush policies, and log retention, providing a complete reference for engineers working with distributed streaming platforms.
This article compares ClickHouse and Elasticsearch, analyzes cost savings, and provides step‑by‑step deployment instructions for Zookeeper, Kafka, Filebeat, and ClickHouse clusters, including configuration details, troubleshooting tips, and practical code snippets for building a scalable analytics pipeline.
This article shares practical development techniques, including how to safely commit Kafka messages within transactions, simplify Redis distributed locks with Redisson, implement an AOP‑based lock annotation for method-level locking, and decouple business logging using asynchronous thread pools and helper utilities.
This article provides a comprehensive overview of Kafka's core components—including brokers, topics, partitions, producers, and consumers—explaining their roles, how they interact within a Kafka cluster, and illustrating the architecture with diagrams while also noting recent version changes and related disclaimer information.
The article details Meituan's massive Kafka deployment—over 15,000 machines handling more than 30 PB of daily data—its performance and management challenges, and the comprehensive application‑layer, system‑layer, and hybrid‑layer optimizations Meituan implemented to reduce read/write latency and improve large‑scale cluster reliability.
This article introduces the core concepts, components, exchange types, reliability mechanisms, ordering, delay, transaction, high‑availability, and load‑balancing strategies of three popular message‑queue systems—RabbitMQ, RocketMQ, and Kafka—while also discussing common challenges such as message ordering, delayed consumption, reliability, idempotence, and backlog handling.
This article explains Kafka's core concepts—including topics, partitions and replicas, log segment storage, leader‑follower mechanics, consumer groups, network threading model, zero‑copy I/O, and the essential role of Zookeeper for broker, topic, consumer, and offset management—providing a comprehensive overview for developers and architects.
This article provides a comprehensive overview of the core components, exchange types, TTL, confirmation mechanisms, dead‑letter queues, and reliability features of RabbitMQ, RocketMQ, and Kafka, along with practical guidance on ordering, delayed consumption, transaction handling, high availability, load balancing, and message deduplication in distributed messaging systems.
The article demonstrates how kafka-go implements consumer groups by creating readers that spawn poll goroutines per partition, explains heartbeat‑driven rebalancing, and compares automatic versus manual offset‑commit strategies, highlighting their impact on throughput, reliability, and message‑processing semantics.
This article provides a comprehensive overview of modern message‑queue systems, detailing RabbitMQ components and exchange types, RocketMQ’s core architecture and high‑availability features, and Kafka’s fundamental concepts, controller mechanisms, and consumer rebalance strategies, while also covering reliability, ordering, and dead‑letter handling techniques.
This article explains the fundamental components and mechanisms of RabbitMQ, RocketMQ, and Kafka—including exchanges, TTL, confirm and return listeners, consumer ACK/NACK, dead‑letter queues, delay and ordered messages, transactional flows, high‑availability setups, load balancing, partitioning, leader election, offset handling, and practical solutions to common messaging challenges.
This article analyzes the challenges of distributed transactions in multi‑node systems and compares various solutions—including 2PC, 3PC, TCC, Saga, local message tables, and MQ‑based approaches—detailing their mechanisms, trade‑offs, and practical implementations with concrete examples from order and cart services.
This article presents a visual full‑link log tracing solution that organizes business logs by execution chain, enabling efficient log collection, dynamic linking, and real‑time visualization to pinpoint issues in complex distributed systems, with a detailed case study from Meituan's content platform.
This article explains Kafka's core architecture, covering the logical concept of topics, physical partitioning and replication, leader‑follower mechanics, consumer groups, log segmentation with zero‑copy I/O, and how Zookeeper manages broker registration, topic metadata, consumer coordination, and offset tracking, while also discussing producer and consumer load‑balancing strategies.
This talk presents Shuhe Technology’s feature platform, detailing its four‑layer architecture, feature storage services, stream‑batch integrated processing, event‑center design, consistency models, and four model‑strategy invocation schemes, illustrating data flows from MySQL through Sqoop, Kafka, Flink, HBase and ClickHouse.
This article details Yelp's end‑to‑end migration of Kafka clusters from EC2‑based brokers to Kubernetes‑based PaaSTA services, covering architectural changes, automated tooling, risk mitigation, rollback procedures, and practical lessons learned from the deployment process.
This article explains the fundamentals of message queues, compares Kafka, RabbitMQ, and RocketMQ in terms of architecture, key concepts, advantages, and drawbacks, and provides practical guidance on selecting the most suitable queue for different application scenarios such as high‑throughput logging, financial transactions, or small‑scale services.
This article provides a comprehensive overview of Kafka's core components—including brokers, topics, partitions, producers, and consumers—explaining their roles, relationships, replication strategies, and how they work together within a Kafka cluster to ensure reliable, scalable message streaming.
This article explains why Stone Docs switched from SLS/ES to ClickHouse for Kubernetes log storage, outlines the four‑stage architecture (collection, transmission, storage, management), compares DaemonSet, network, and SideCar collection methods, and provides concrete ClickHouse table definitions and Fluent‑Bit configurations for a production‑grade logging pipeline.
This article explains how to create a lightweight, plug‑in‑style microservice flow orchestration system using Docker‑installed Nacos for configuration, three SpringBoot services with Kafka integration, and dynamic topic management to enable hot‑swapable data pipelines without heavy frameworks.
This article visually explains Kafka's core concepts—including producers, consumers, topics, partitions, consumer groups, and cluster architecture—so readers can clearly understand how messages flow, are stored, and remain fault‑tolerant within a distributed streaming system.
Songguo Mobility’s data‑center team migrated from a fragmented Impala‑Kudu‑ClickHouse stack to a unified StarRocks‑based real‑time OLAP architecture, iterating through three versions to solve scalability, latency, and maintenance challenges while supporting minute‑level dashboards for orders and vehicle analytics.
This article thoroughly examines Kafka's broker‑side network architecture, tracing its evolution from a simple sequential model to a high‑performance, event‑driven Reactor design using Java NIO, and provides practical tuning guidance for achieving optimal throughput and latency.
This article explains the fundamentals of message middleware, covering its architecture, components, communication models, advantages, common protocols, major open‑source solutions, detailed feature comparisons, and practical guidance for selecting the most suitable message queue based on system scale, business needs, performance, high availability, and operational considerations.
This article introduces Faust, an open‑source Python library that brings Kafka Streams‑style stream processing to Python, covering its features, installation, a step‑by‑step example, typed data models, and how to run real‑time data pipelines with async support.
This article explains common Kafka message‑loss and duplicate‑consumption issues, introduces Alibaba Cloud's fully managed Kafka Retrieval Component, and provides step‑by‑step guidance—including enabling the service, using Tablestore for multi‑index and SQL searches—to help engineers quickly locate and verify missing or duplicated messages.
This article provides a comprehensive introduction to three major message‑queue systems—RabbitMQ, RocketMQ, and Kafka—explaining their core components, exchange types, delivery guarantees, confirmation mechanisms, dead‑letter handling, high‑availability designs, load‑balancing strategies, and common troubleshooting techniques.
This article explains how to build a lightweight, plug‑and‑play microservice flow orchestration system using Docker‑installed Nacos, SpringBoot services with Kafka integration, and dynamic configuration listeners to adjust data pipelines without downtime.
During a pre‑GIAC interview, Shopee data engineer Deng Lin discusses the evolution of data lakes and warehouses, lakehouse integration, big‑data technology choices, real‑time processing with Flink and Kafka, and offers career advice for newcomers to the big‑data field.
Yelp migrated its massive Kafka deployment from EC2 to a Kubernetes‑based PaaSTA platform, introducing a custom Kafka operator and Cruise Control to cut provisioning time, simplify upgrades, improve lifecycle management, and reduce reliance on Puppet, while maintaining high‑throughput data pipelines.
This article explains how to solve cross‑server WebSocket communication in a distributed Java application by using a message queue (Redis or Kafka) to broadcast messages, storing user‑node mappings, and providing complete Spring configuration, code examples, and a demo page for end‑to‑end testing.
This article thoroughly examines Kafka Broker's network architecture and request‑handling pipeline, exploring sequential, multi‑threaded, and Reactor‑based designs, detailing Acceptor and Processor threads, NIO mechanisms, and offering practical tuning tips to achieve high concurrency and performance.
This article explains how a Kafka client determines which broker listener to use when a cluster has multiple listeners, detailing the metadata update flow, server‑side filtering by listener name, and the impact of mixed listener names in the bootstrap.servers list.
This article explains how to create a plug‑and‑play microservice flow orchestration using Docker‑based Nacos, three SpringBoot services, and Kafka, covering installation, configuration, dynamic topic management, and runtime consumer updates for flexible data pipelines.
This article provides a diagram‑driven walkthrough of Kafka’s fundamental concepts—including topics, partitions, producers, consumers, consumer groups, and cluster architecture—explaining how messages flow, are stored, and achieve reliability and ordering within a distributed streaming system.
This article explains what a Message Queue (MQ) is, why it is used for traffic shaping, system decoupling, and asynchronous processing, reviews popular MQ products such as Kafka, ActiveMQ, RocketMQ and RabbitMQ, and details RabbitMQ's core concepts and advanced features like dead‑letter, delayed, idempotent, priority, and lazy queues.
This article analyzes the shortcomings of the traditional poll‑then‑process loop in Kafka consumers, explains configuration pitfalls such as max.poll.interval.ms, and proposes a more robust architecture using work queues, a poller, an executor, and an offset manager to achieve at‑most‑once, at‑least‑once, and exactly‑once processing guarantees.
This article visually breaks down Kafka’s fundamental concepts—including topics, partitions, producers, consumers, consumer groups, and cluster architecture—so readers can grasp how messages flow, are stored, and achieve load balancing and ordering within a distributed streaming platform.
Zuoyebang’s data team replaced fragmented, slow query solutions with Apache Doris, building a unified real‑time data warehouse that dramatically cut query latency from hours to seconds, streamlined data modeling, and improved reliability across diverse business scenarios, while integrating with Flink, Kafka, and ES via a unified API.
This article explains how Apache Flink achieves end‑to‑end exactly‑once semantics by using source replay support, checkpoint‑based snapshots, asynchronous incremental checkpoints, and two‑phase commit sinks, and describes the interaction with external systems such as Kafka to ensure transactional writes.
This article explains why Kafka suffers from broker‑side load imbalance, walks through manual replica migration examples, and then details how Cruise Control automates load balancing, supports resource‑group targeting, leader‑replica dispersion, and provides step‑by‑step deployment instructions.
This article explains the fundamental concepts of Apache Kafka, covering its role as a streaming platform, the producer‑consumer model, how topics and partitions work, consumer groups for load balancing, message ordering, replication with leaders and followers, and the coordination role of ZooKeeper.
This article explains the fundamental concepts of Apache Kafka, including producers, consumers, topics, partitions, consumer groups, and the role of ZooKeeper in a clustered architecture, while illustrating how messages are stored, routed, and ordered for reliable, scalable stream processing.
The article recounts a production incident where a Kafka‑based consumer in a finance microservice hit thread‑pool exhaustion and slow‑query alerts, analyzes the root causes of async processing and bulk message bursts, and outlines a three‑phase remediation that includes data repair, switching to synchronous consumption, and request‑level batching to prevent future failures.
Kafka’s server‑side load imbalance, caused by static replica placement on broker disks, makes manual replica migration infeasible at scale, but Cruise Control automates metric collection, analysis, and execution of fine‑grained rebalance plans—including broker de‑commissioning and leader dispersion—allowing large clusters to expand and operate efficiently.
This article compares Redis Streams with Kafka, highlighting Redis's simplicity, performance, and built‑in features like consumer groups, while discussing when each solution is appropriate for different project scales and requirements.
This article introduces and compares four widely used open-source distributed message-queue systems—Kafka, ActiveMQ, RabbitMQ, and RocketMQ—detailing their architectures, deployment requirements, features, high-availability solutions, advantages, disadvantages, and provides guidance for selecting the appropriate middleware in large-scale applications.
Vipshop’s Dragonfly log platform evolved from a costly 260‑node Elasticsearch cluster to a ClickHouse‑based architecture that uses a unified JSON format, vfilebeat ingestion, Flink parsing, and MergeTree storage to achieve high‑throughput writes, fast vectorized queries, flexible TTL management, and dramatically lower operational expenses.
This guide introduces Sentry's open‑source error and performance monitoring platform, outlines how to set up and extend the service, and details its event‑processing architecture, including Relay, Redis caching, Symbolicator, and alert handling via Celery and Kafka.
This article visually breaks down Kafka’s core concepts—including topics, partitions, producers, consumers, and cluster architecture—explaining how messages flow, how consumer groups achieve load balancing, and how ZooKeeper coordinates leaders and followers, providing a clear, illustrated guide for developers working with distributed streaming systems.
This article shares G7 Tech’s practical experience of exposing Kafka to the public internet, covering encryption, AAA, three authentication schemes, listener configuration, scaling for massive topics with Kubernetes, storage optimization, and integration with the gmq management platform and Kafka‑REST.
