Tagged articles
log aggregation
51 articles · Page 1 of 1
A comprehensive guide to deploying Loki with Promtail/Alloy for centralized logging, covering architecture decisions, label design, single-binary deployment with object storage, LogQL query patterns, monitoring, troubleshooting, and migration from Promtail to Alloy.
AlloyGrafanaLogQL
0 likes · 54 min read

This tutorial teaches five native kubectl techniques for rapidly aggregating and filtering Kubernetes logs across multiple pods, including label selectors, time-based filtering, pod-prefix scripting, the kubetail tool for colored multi-pod streaming, and the --previous flag for crashed pods, plus a ready-to-use troubleshooting template.
CrashLoopBackOffKuberneteskubectl
0 likes · 8 min read

Loki + Promtail provides a lightweight log aggregation solution that indexes only labels, cutting storage and memory usage to about one‑fifth of ELK, and the article walks through deployment, configuration, best‑practice label design, multi‑tenant setup, performance tuning, and real‑world case studies.
LokiPromtailcloud native
0 likes · 40 min read

This guide walks through preparing a Linux host, downloading Loki v3.6.10, configuring its YAML file, setting up a systemd service, deploying Promtail for log collection, and integrating Grafana for visualization, all using a unified /app/soft directory structure.
CentOSGrafanaLinux
0 likes · 10 min read

This comprehensive 2025 guide compares ELK, EFK, and Loki across architecture, deployment complexity, storage cost, query performance, feature completeness, high‑availability, and real‑world case studies, helping teams of any size choose the most cost‑effective and operationally suitable log collection stack.
EFKLokielk
0 likes · 37 min read

This article explains how to centralize and analyze massive multi‑cloud log data stored in object storage by moving AWS S3 logs into Alibaba Cloud Log Service (SLS) using dual‑mode file discovery, SQS event‑driven import, elastic scaling, and pre‑ingestion processing to achieve low latency, high reliability, and cost efficiency.
AWS S3Data IngestionElastic Scaling
0 likes · 12 min read

This comprehensive guide walks you through the challenges of large‑scale log collection, real‑time processing, storage optimization, and visualization, detailing practical configurations for Flume, Logstash, Elasticsearch, Kibana, Filebeat, Kafka, Kubernetes, and future AIOps integrations to build a reliable, cost‑effective distributed logging system.
FlumeKafkaObservability
0 likes · 24 min read

This guide explains why log aggregation is needed in micro‑service environments, shows how to set up Graylog using Docker‑Compose, configure its inputs, and integrate it with a Spring Boot application via Logback‑GELF, then demonstrates basic log search queries.
DockerGraylogLogging
0 likes · 8 min read

Learn step‑by‑step how to set up Graylog using Docker‑Compose, configure its inputs, and connect a Spring Boot application via Logback‑GELF, enabling centralized log aggregation and searchable queries across multiple service instances in a micro‑services environment.
DockerGraylogLogback
0 likes · 8 min read

Loki, an open‑source log aggregation system from Grafana Labs, integrates tightly with Prometheus and Grafana, stores logs efficiently using object storage, offers a simple label‑based model, and provides cost‑effective, high‑performance logging for cloud‑native environments while outlining its architecture, usage, configuration, advantages, limitations, and retention policies.
GrafanaLokiObservability
0 likes · 10 min read

This article explains how to use Filebeat for log shipping, configure its YAML files, deploy Graylog with Docker and Elasticsearch, and integrate logging into Spring Boot applications, providing step‑by‑step commands, code examples, and best‑practice recommendations for centralized log management.
DockerElasticsearchGraylog
0 likes · 20 min read

This step‑by‑step guide shows how to install Loki, configure its YAML file, set up Promtail to ship logs, install Grafana, add Loki as a data source, and use LogQL to query logs—including collecting Nginx JSON logs—on a Rocky Linux system.
GrafanaLogQLLoki
0 likes · 10 min read

This article explains why centralized log collection is essential for multi‑environment services, introduces Graylog as a lightweight alternative to ELK, details Filebeat's role and workflow, provides configuration examples, shows how to deploy both Filebeat and Graylog via Docker or packages, and demonstrates integration with Spring Boot and log search techniques.
DockerGraylogelk
0 likes · 20 min read

This tutorial explains how to deploy Graylog using Docker‑Compose for centralized log aggregation in a microservices environment and shows step‑by‑step integration of Graylog with a Spring Boot application via Logback‑GELF, including configuration, code examples, and basic log search queries.
Docker ComposeGELFGraylog
0 likes · 8 min read


ITPUBDec 24, 2023 · Backend Development
This article explains how Kafka serves as a high‑throughput, durable messaging system, a reliable storage layer, a log‑aggregation hub, a stream‑processing engine, and a core component for CDC, system migration, monitoring, and event‑sourcing architectures.
CDCKafkaevent-sourcing
0 likes · 9 min read

This article examines the challenges of building a high‑performance log service for distributed systems, compares the traditional ELK stack with the Tianyan platform, details Tianyan's architecture—including ingest, storage, and consumer components, SDK and Minos collection methods, high‑throughput transmission with Disruptor and Bigpipe, log retrieval, resource isolation, dynamic cleaning, and best‑practice recommendations.
BigpipeDisruptorLogging
0 likes · 27 min read

This article analyzes the challenges of logging in distributed services, compares the traditional ELK stack with Baidu's Tianyan platform, and details Tianyan's architecture, data collection, high‑throughput transmission, storage, retrieval, resource isolation, dynamic cleanup, and best‑practice recommendations, complete with code examples and performance insights.
ElasticsearchJavabig data
0 likes · 30 min read

This guide introduces two lightweight Kubernetes log‑tailing tools, Kubetail and Stern, explains their installation on various platforms, demonstrates common usage patterns and command‑line options, and provides practical examples for aggregating and filtering logs across multiple pods and containers.
DevOpsKubernetescloud native
0 likes · 10 min read

This article explains why Loki is a lightweight, cloud‑native log aggregation solution, outlines its advantages and supported storage backends, compares log collectors, details Loki's indexing and query mechanisms, and provides step‑by‑step instructions for deploying Loki in Kubernetes with all‑in‑one, read/write, and microservice modes.
Cloud Native MonitoringGrafanaKubernetes
0 likes · 15 min read

This article describes the design, implementation, and evolution of Heimdall, an exception‑statistics platform built on Kafka, Flink, and HBase that provides minute‑level anomaly aggregation, stack trace querying, and integration with release and alerting workflows to improve service reliability across thousands of micro‑services.
Exception MonitoringKafkaObservability
0 likes · 14 min read

This article provides a detailed comparison of nine popular log management solutions—Filebeat, Graylog, LogDNA, ELK Stack, Grafana Loki, Datadog, Logstash, Fluentd, and Splunk—covering their core features, pricing models, advantages, and drawbacks to help readers choose the right tool for centralized logging.
cloudelklog aggregation
0 likes · 13 min read

This guide introduces two lightweight Kubernetes log‑tailing utilities, Kubetail and Stern, explaining their installation on various platforms, core command‑line options, and practical usage examples for aggregating and color‑coding logs from multiple pods and containers, offering a simpler alternative to heavyweight logging stacks.
Kubernetesclikubetail
0 likes · 10 min read

This article describes how Meituan built Logan, a high‑performance, end‑to‑end real‑time logging platform for mobile, web, mini‑programs and IoT, covering its background, architecture, data collection, processing, consumption, monitoring, deployment strategies, achieved results and future roadmap.
ElasticsearchFlinkKafka
0 likes · 15 min read

This tutorial explains how to set up Graylog for centralized log aggregation using Docker Compose, configure its inputs, and integrate a Spring Boot application with Graylog via Logback GELF, including search syntax examples for effective log analysis.
DockerGELFGraylog
0 likes · 7 min read

This article introduces Grafana Loki, an open‑source, horizontally scalable, highly available log aggregation system optimized for Kubernetes and Prometheus, covering its core concepts, architecture, component roles, deployment steps, configuration examples, and practical usage within Grafana.
GrafanaKubernetesLoki
0 likes · 18 min read

This guide introduces Loki, the open‑source, horizontally scalable log aggregation system optimized for Prometheus and Kubernetes, covering its core concepts, architecture, components, deployment steps, Grafana integration, label‑based indexing, and best practices for handling dynamic and high‑cardinality tags.
GrafanaKubernetesLoki
0 likes · 19 min read

This article explains the motivations for selecting Grafana Loki instead of traditional ELK/EFK stacks, introduces Loki's core concepts and architecture, details component roles, provides step‑by‑step deployment of Promtail and Loki, and demonstrates how to configure and query logs in Grafana while addressing label indexing, dynamic tags, high‑cardinality challenges, and query performance.
GrafanaKubernetesLoki
0 likes · 18 min read

This article explains how to build a terabyte‑level log monitoring platform for micro‑service environments by unifying log collection with FileBeat, enriching observability through Elastic APM, processing streams via Kafka Streams, and visualizing metrics with Grafana and Kibana, while addressing cost‑effective filtering and retention strategies.
ELK stackGrafanaKafka Streams
0 likes · 8 min read

This guide walks you through setting up a Loki logging cluster for production, covering the native architecture, key configuration differences, storage with boltdb‑shipper on S3, Redis caching, ruler setup, and adapting the Docker‑Compose deployment to Kubernetes.
ConfigurationKubernetesLoki
0 likes · 9 min read

This article explains the motivations behind choosing Grafana Loki over traditional ELK/EFK stacks for container‑cloud logging, detailing its lightweight design, cost advantages, simple architecture, and how its components—Distributor, Ingester, and Querier—work together to provide scalable, efficient log aggregation and querying.
LokiPrometheuslog aggregation
0 likes · 8 min read

This article explains Loki's low‑index overhead, concurrent query handling, tag‑based indexing, component roles, read/write paths, step‑by‑step installation of Promtail and Loki, label matching techniques, dynamic‑tag handling, high‑cardinality concerns, and query optimization strategies for cloud‑native log aggregation.
LokiObservabilityPromtail
0 likes · 13 min read

This article explores the challenges of log aggregation in micro‑service architectures, introduces a globally unique log identifier (logid) with its required properties, compares various logid generation schemes, and presents end‑to‑end solutions for log distribution, aggregation, and analysis using custom tools such as ylog and watcher.
distributed systemslog aggregationlog analysis
0 likes · 26 min read

This article explains Loki's advantages over Elasticsearch, including low indexing overhead, concurrent query processing with caching, seamless integration with Prometheus and Grafana, detailed architecture components, installation steps, label handling, high‑cardinality challenges, and best practices for efficient log management.
ElasticsearchGrafanaLoki
0 likes · 15 min read

This guide walks you through setting up a Loki logging cluster in production, covering native configurations, extended storage and cache settings, Kubernetes deployment, and practical code examples to simplify the process for newcomers.
KubernetesLokiS3-storage
0 likes · 8 min read

Learn step‑by‑step how to design and deploy a billion‑scale log collection and analysis platform using the ELK stack—Filebeat, Kafka, Logstash, Elasticsearch, and Kibana—covering architecture, configuration, installation, and best practices for high‑availability and performance.
ElasticsearchKibanaLogstash
0 likes · 14 min read

This article introduces Loki, Grafana’s lightweight, horizontally scalable log aggregation system, compares it with the EFK stack, explains Promtail, LogQL query language, alerting, and how Loki integrates with Grafana and Prometheus for unified metrics and logs, highlighting its low‑resource, cloud‑native advantages.
LokiObservabilitycloud native
0 likes · 8 min read

This guide walks you through Loki's architecture, advantages, installation via Docker Compose, configuration of Promtail, and step‑by‑step commands to set up a high‑availability, multi‑tenant log aggregation system integrated with Grafana.
Docker ComposeGrafanaLoki
0 likes · 7 min read

This article provides a comprehensive overview of Apache Flume, detailing its purpose as a distributed log aggregation system, explaining its core components such as sources, channels, and sinks, and illustrating its architecture, multi‑agent setups, and key features like reliability, scalability, compression, and monitoring.
Data IngestionFlumelog aggregation
0 likes · 9 min read

This article explains Loki's origins, data model, LogQL query language, low‑cost storage design, and the full read‑write architecture—including Distributor, Ingester, Querier, and QueryFrontend—showing how it solves the shortcomings of traditional Elasticsearch‑based logging solutions and integrates tightly with Prometheus monitoring.
LogQLLokiObservability
0 likes · 21 min read

Loki 2.0.0 introduces major enhancements such as a revamped LogQL pipeline, native Prometheus‑style alerts, and simplified storage with boltdb‑shipper, delivering a more resource‑efficient, scalable log aggregation solution for Kubernetes environments.
KubernetesLogQLLoki
0 likes · 3 min read

This article explains the motivations behind choosing Loki over traditional ELK/EFK stacks for container‑cloud logging, outlines its cost‑effective design, describes its simple architecture and components such as Distributor, Ingester, and Querier, and highlights its scalability and seamless integration with Prometheus.
ELK alternativeLokiObservability
0 likes · 8 min read

An in‑depth look at Bilibili’s multi‑layer monitoring overhaul, detailing the shift from a monolithic Zabbix setup to micro‑service‑based ELK, Dapper, Misaka, Traceon and Lancer systems, and how layered observability improves fault detection across business, application, and infrastructure levels.
Distributed TracingObservabilitylog aggregation
0 likes · 10 min read

To cope with Meituan Hotel’s exploding micro‑service complexity, the infrastructure team built the Satellite System—combining MTrace and a selective, zero‑intrusion Log4j2‑based logging pipeline that streams enriched logs through Kafka, Storm, Redis and Elasticsearch, delivering second‑level trace‑log queries and six‑month retention, dramatically speeding up debugging.
Distributed TracingElasticsearchKafka
0 likes · 11 min read


21CTODec 3, 2017 · Operations
This article explains the core components of the ELK stack, compares three typical deployment architectures, and provides practical solutions for multiline log merging, timestamp correction, and module‑based log filtering using Filebeat, Logstash, Kafka, and Kibana.
ElasticsearchKibanaLogstash
0 likes · 10 min read

This article introduces CAT, an open‑source Java‑based distributed real‑time monitoring platform, detailing its design goals, architecture, message processing pipeline, logging instrumentation, API, real‑time analysis, report modeling, storage challenges, and key takeaways for building highly available, scalable monitoring solutions.
Distributed Monitoringlog aggregationoperations
0 likes · 13 min read

This article introduces Apache Flume, explains its design goals of reliability, scalability, manageability and extensibility, outlines core concepts and architecture, provides step‑by‑step configuration using the first mode, demonstrates integration with Zookeeper, Kafka and a shell script, and shows how to launch and verify the agent.
Apache FlumeKafka Integrationbig data
0 likes · 7 min read

This article explains Hadoop 2.0's built‑in MRv2 log collection mechanism, detailing job‑run and task‑run logs, their generation steps, log aggregation, and the role of the JobHistory Server for centralized analysis.
HadoopJobHistoryMapReduce
0 likes · 8 min read


21CTOMay 16, 2016 · Operations
This article explains a practical architecture for aggregating logs from distributed Docker containers by employing Flume NG as a lightweight log collector, Kafka as a high‑throughput message bus, and custom sinks to store logs per service, module and day with low latency and minimal resource impact.
DockerFlumeKafka
0 likes · 17 min read

This article provides a comprehensive overview of Apache Flume, covering its design goals, core components, deployment architecture, configuration patterns, and step‑by‑step instructions for integrating Flume with Zookeeper and Kafka to collect and forward massive log data.
Apache FlumeKafkaZooKeeper
0 likes · 6 min read


21CTOMar 17, 2016 · Operations
This article explains Vipshop’s multi‑layer monitoring architecture, detailing system‑level metrics, application‑level tracing with the Mercury platform, and business‑level KPI dashboards, while describing the data pipelines that collect, process, and alert on distributed logs to ensure reliable operations.
Vipshopdistributed systemslog aggregation
0 likes · 4 min read

Vipshop’s three‑tier monitoring system—covering system, application (Mercury), and business layers—collects and analyzes logs from distributed components, providing real‑time metrics, slow‑call detection, error tracing, and configurable alerts to help engineers quickly pinpoint and resolve performance issues.
APMalertingdistributed systems
0 likes · 4 min read
