Skip to content
Tagged articles

Log Collection

117 articles · Page 2 of 2
21CTO
21CTO
Sep 2, 2018 · Operations

How We Built a Real‑Time Log Collection System with Go, Kafka, and Etcd

This article examines the shortcomings of a legacy log‑collection setup, introduces a streamlined real‑time architecture that eliminates per‑machine Hadoop clients, centralizes configuration via Etcd with a web UI, and details the Go implementation—including configuration design, Etcd watching, Kafka integration, tail management, and per‑second rate limiting—complete with code snippets and performance graphs.

Log CollectionReal-timeetcd
0 likes · 16 min read
How We Built a Real‑Time Log Collection System with Go, Kafka, and Etcd
Snowball Engineer Team
Snowball Engineer Team
Mar 23, 2018 · Big Data

Redesigning Snowball's Log Collection Architecture During Hadoop Cluster Expansion

The article details Snowball's challenges with a saturated CDH Hadoop cluster, outlines the limitations of the original Kafka‑based log pipeline, and explains how a comprehensive redesign using FlumeNG, Spillable Memory Channels, and custom HDFS sinks resolves latency, data loss, and high‑load issues while supporting future growth.

Cluster MigrationFlumeNGHadoop
0 likes · 6 min read
Redesigning Snowball's Log Collection Architecture During Hadoop Cluster Expansion
Efficient Ops
Efficient Ops
Mar 7, 2018 · Operations

Mastering Log Collection: From Daily Ops to the ELK Stack

This article explores the everyday challenges of operations teams handling system, access, runtime, error, and business logs, outlines the pain points of log collection and standardization, and provides a comprehensive guide to implementing the ELK (Elastic) stack—including Elasticsearch, Logstash, and Kibana—for effective monitoring and analysis.

KibanaLog CollectionLogstash
0 likes · 13 min read
Mastering Log Collection: From Daily Ops to the ELK Stack
Meitu Technology
Meitu Technology
Dec 19, 2017 · Industry Insights

Inside Meitu’s In‑House Log Collection System Arachnia: Design, Challenges, and Core Mechanisms

This article introduces Meitu’s self‑developed log collection system Arachnia, explaining why a custom solution was needed for massive server‑side user‑behavior logs, the key requirements such as reliability and real‑time throughput, and the core architectural mechanisms that address those challenges.

ArachniaLog CollectionMeitu
0 likes · 2 min read
Inside Meitu’s In‑House Log Collection System Arachnia: Design, Challenges, and Core Mechanisms
dbaplus Community
dbaplus Community
Oct 10, 2017 · Operations

How to Build Effective Service Monitoring: Principles, Practices, and Technical Implementation

This article explains why service monitoring is essential for large‑scale microservice environments, outlines design principles, core monitoring components, dependency mapping, call‑chain analysis, capacity planning, root‑cause analysis, and presents a practical technical architecture for implementing robust monitoring solutions.

Distributed TracingLog Collectioncapacity planning
0 likes · 12 min read
How to Build Effective Service Monitoring: Principles, Practices, and Technical Implementation
21CTO
21CTO
Sep 28, 2017 · Operations

Master Real-Time Log Collection with LogHub: Strategies for E‑Commerce Platforms

This article explains how LogHub enables real-time log collection and unified management for an e‑commerce takeout platform, covering operational challenges, logstore configuration, user promotion tracking, server and client logging methods, and network access options.

Data IngestionLog CollectionLogHub
0 likes · 9 min read
Master Real-Time Log Collection with LogHub: Strategies for E‑Commerce Platforms
Programmer DD
Programmer DD
Aug 25, 2017 · Backend Development

How to Build a Logback‑ELK Log Collection Server for Spring MVC

This guide walks you through installing and configuring the ELK stack (Elasticsearch, Logstash, Kibana), setting up the required environment, creating Logback and Logstash configurations, and integrating them with a Spring MVC application to collect and visualize Java logs in Kibana.

JavaLog CollectionLogback
0 likes · 13 min read
How to Build a Logback‑ELK Log Collection Server for Spring MVC
Java Backend Technology
Java Backend Technology
Mar 2, 2017 · Operations

Unlock ELK Beats: Filebeat, Metricbeat, Packetbeat, Winlogbeat & Heartbeat

Beats are lightweight data shippers in the ELK Stack, and this guide details the purpose, key features, and integration of Filebeat for log collection, Metricbeat for system and service metrics, Packetbeat for network traffic analysis, Winlogbeat for Windows event logs, and Heartbeat for service uptime monitoring.

BeatsLog CollectionSystem Monitoring
0 likes · 8 min read
Unlock ELK Beats: Filebeat, Metricbeat, Packetbeat, Winlogbeat & Heartbeat
dbaplus Community
dbaplus Community
Sep 6, 2016 · Big Data

Choosing the Right Log Collection Framework for Massive Data Streams

This article reviews major open‑source log collection tools—Chukwa, Scribe, Flume, Logstash, Kafka, and TT—examining their architectures, strengths, and limitations to help engineers select the most suitable solution for high‑volume, low‑latency data pipelines.

Apache FlumeKafkaLog Collection
0 likes · 13 min read
Choosing the Right Log Collection Framework for Massive Data Streams
Architect
Architect
May 16, 2016 · Operations

Centralized Log Collection for Distributed Docker Services Using Flume and Kafka

This article presents a practical architecture for centrally collecting dispersed logs from Docker‑based services in a distributed environment by leveraging Flume NG as a non‑intrusive log agent, Kafka as a high‑throughput message bus, and custom sinks to partition logs by service, module, and day.

DockerKafkaLog Collection
0 likes · 15 min read
Centralized Log Collection for Distributed Docker Services Using Flume and Kafka
Architect
Architect
Feb 18, 2016 · Cloud Native

Collecting Docker Container Logs with Flume: Strategies and Implementation

This article explains how to capture Docker container logs, discusses the challenges of multi‑line log correlation, and presents two approaches—client‑side parsing and server‑side parsing—along with a concrete Flume customization using a DockerLog Java bean.

DockerFlumeJava
0 likes · 7 min read
Collecting Docker Container Logs with Flume: Strategies and Implementation
Baidu Tech Salon
Baidu Tech Salon
Mar 10, 2014 · Cloud Computing

Building a Scalable App Engine Platform: Architecture, Open‑Source Tools & Best Practices

This article provides a comprehensive overview of App Engine as a PaaS solution, analyzes leading platforms such as CloudFoundry and SAE, outlines architectural requirements, and presents practical implementation guidance using open‑source tools like Nginx, Scribe, and Storm for elastic, scalable cloud services.

App EngineElastic ScalingLog Collection
0 likes · 20 min read
Building a Scalable App Engine Platform: Architecture, Open‑Source Tools & Best Practices