This article explains how Kafka's Schema Registry manages schema evolution and compatibility types—backward, forward, transitive, and full—using Avro schemas, demonstrates the impact of field additions or deletions on producers and consumers, and shows how to change a topic's compatibility setting via REST API.
This article provides a comprehensive English guide to Kafka's producer component, covering its end‑to‑end workflow, detailed steps such as record creation, serialization, partitioning, batching, sender thread, custom data structures, send modes, common exceptions, and essential configuration parameters for optimal performance.
This article chronicles the four‑year evolution of Chehaoduo Group’s Kafka ecosystem—from its initial role as a simple data‑ingestion layer to becoming the core of the company’s large‑scale data pipeline—detailing cluster management, upgrade strategies, multi‑cluster deployment, AVRO schema handling, SDK development, and operational lessons learned.
Apache Avro is a high‑performance binary data serialization system originating from Hadoop that uses JSON‑defined schemas to enable compact storage, efficient network transfer, container file formats for MapReduce, and RPC communication without requiring code generation or explicit field numbers.
This article compares CSV, JSON, Parquet, and Avro file formats, outlining their structures, advantages, and drawbacks, and explains how Apache Spark supports each format for efficient big‑data storage and processing.
This article details how Vimur, a Kafka‑based real‑time data pipeline, addresses the challenges of service splitting, database sharding, data migration, and synchronization by using CDC, a unified Avro format, and a change distribution platform to support search indexing, cache refresh, and reactive architectures.
This article details Ctrip's design and implementation of a high‑throughput, low‑latency user data collection platform that leverages Java NIO, Netty, and a custom Kafka‑based messaging layer, covering architecture, encryption, compression, disaster‑recovery, performance testing, and downstream analytics products.
This article explains how to use the open‑source Kiji framework together with HBase, Avro, and MapReduce to build a scalable, entity‑centric real‑time recommendation system that can instantly refresh suggestions based on user context and recent interactions.
