Building a High-Throughput Kubernetes-Based Log Processing System at Dada
The article describes how Dada rebuilt its log processing pipeline using Kubernetes mixed deployment, Filebeat for automated collection, Storm for efficient parsing, and Elasticsearch cold/hot nodes to handle over 130 billion daily log entries and 300TB storage.
The article outlines the evolution of Dada's log system after merging with JD Daojia in 2016, highlighting the limitations of the original ELK stack—manual Flume configuration, lack of log formatting, and chaotic Elasticsearch storage.
To address these issues, Dada standardized log formats and replaced Flume with Filebeat for automated log collection, tagging logs by source and sending them to appropriate Kafka topics.
Log parsing was migrated from Logstash to a custom Storm topology, where a logParserBolt routes logs based on tags to specific parsing functions, improving efficiency and reducing maintenance overhead.
Elasticsearch was reorganized with a hot/warm node architecture; hot nodes handle recent logs while warm nodes store older data, and Kubernetes was used to deploy warm nodes, utilizing otherwise idle CPU resources.
By rewriting the parsing logic in Go and running it on Kubernetes warm nodes, Dada replaced the Storm cluster, achieving sufficient processing power for peak loads of over 130 billion log entries per day and 300 TB total storage.
The article concludes with ongoing research into ClickHouse as a potential replacement for Elasticsearch to further enhance log analysis capabilities.
Dada Group Technology
Sharing insights and experiences from Dada Group's R&D department on product refinement and technology advancement, connecting with fellow geeks to exchange ideas and grow together.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.