This comprehensive 2025 guide compares ELK, EFK, and Loki across architecture, deployment complexity, storage cost, query performance, feature completeness, high‑availability, and real‑world case studies, helping teams of any size choose the most cost‑effective and operationally suitable log collection stack.
This article explains how to centralize and analyze massive multi‑cloud log data stored in object storage by moving AWS S3 logs into Alibaba Cloud Log Service (SLS) using dual‑mode file discovery, SQS event‑driven import, elastic scaling, and pre‑ingestion processing to achieve low latency, high reliability, and cost efficiency.
This comprehensive guide walks you through the challenges of large‑scale log collection, real‑time processing, storage optimization, and visualization, detailing practical configurations for Flume, Logstash, Elasticsearch, Kibana, Filebeat, Kafka, Kubernetes, and future AIOps integrations to build a reliable, cost‑effective distributed logging system.
This guide explains why log aggregation is needed in micro‑service environments, shows how to set up Graylog using Docker‑Compose, configure its inputs, and integrate it with a Spring Boot application via Logback‑GELF, then demonstrates basic log search queries.
Learn step‑by‑step how to set up Graylog using Docker‑Compose, configure its inputs, and connect a Spring Boot application via Logback‑GELF, enabling centralized log aggregation and searchable queries across multiple service instances in a micro‑services environment.
Loki, an open‑source log aggregation system from Grafana Labs, integrates tightly with Prometheus and Grafana, stores logs efficiently using object storage, offers a simple label‑based model, and provides cost‑effective, high‑performance logging for cloud‑native environments while outlining its architecture, usage, configuration, advantages, limitations, and retention policies.
This article explains how to use Filebeat for log shipping, configure its YAML files, deploy Graylog with Docker and Elasticsearch, and integrate logging into Spring Boot applications, providing step‑by‑step commands, code examples, and best‑practice recommendations for centralized log management.
This step‑by‑step guide shows how to install Loki, configure its YAML file, set up Promtail to ship logs, install Grafana, add Loki as a data source, and use LogQL to query logs—including collecting Nginx JSON logs—on a Rocky Linux system.
This article explains why centralized log collection is essential for multi‑environment services, introduces Graylog as a lightweight alternative to ELK, details Filebeat's role and workflow, provides configuration examples, shows how to deploy both Filebeat and Graylog via Docker or packages, and demonstrates integration with Spring Boot and log search techniques.
This tutorial explains how to deploy Graylog using Docker‑Compose for centralized log aggregation in a microservices environment and shows step‑by‑step integration of Graylog with a Spring Boot application via Logback‑GELF, including configuration, code examples, and basic log search queries.
This article explains how Kafka serves as a high‑throughput, durable messaging system, a reliable storage layer, a log‑aggregation hub, a stream‑processing engine, and a core component for CDC, system migration, monitoring, and event‑sourcing architectures.
This article examines the challenges of building a high‑performance log service for distributed systems, compares the traditional ELK stack with the Tianyan platform, details Tianyan's architecture—including ingest, storage, and consumer components, SDK and Minos collection methods, high‑throughput transmission with Disruptor and Bigpipe, log retrieval, resource isolation, dynamic cleaning, and best‑practice recommendations.
This article analyzes the challenges of logging in distributed services, compares the traditional ELK stack with Baidu's Tianyan platform, and details Tianyan's architecture, data collection, high‑throughput transmission, storage, retrieval, resource isolation, dynamic cleanup, and best‑practice recommendations, complete with code examples and performance insights.
This guide introduces two lightweight Kubernetes log‑tailing tools, Kubetail and Stern, explains their installation on various platforms, demonstrates common usage patterns and command‑line options, and provides practical examples for aggregating and filtering logs across multiple pods and containers.
This article explains why Loki is a lightweight, cloud‑native log aggregation solution, outlines its advantages and supported storage backends, compares log collectors, details Loki's indexing and query mechanisms, and provides step‑by‑step instructions for deploying Loki in Kubernetes with all‑in‑one, read/write, and microservice modes.
This article describes the design, implementation, and evolution of Heimdall, an exception‑statistics platform built on Kafka, Flink, and HBase that provides minute‑level anomaly aggregation, stack trace querying, and integration with release and alerting workflows to improve service reliability across thousands of micro‑services.
This article provides a detailed comparison of nine popular log management solutions—Filebeat, Graylog, LogDNA, ELK Stack, Grafana Loki, Datadog, Logstash, Fluentd, and Splunk—covering their core features, pricing models, advantages, and drawbacks to help readers choose the right tool for centralized logging.
This guide introduces two lightweight Kubernetes log‑tailing utilities, Kubetail and Stern, explaining their installation on various platforms, core command‑line options, and practical usage examples for aggregating and color‑coding logs from multiple pods and containers, offering a simpler alternative to heavyweight logging stacks.
This article describes how Meituan built Logan, a high‑performance, end‑to‑end real‑time logging platform for mobile, web, mini‑programs and IoT, covering its background, architecture, data collection, processing, consumption, monitoring, deployment strategies, achieved results and future roadmap.
This tutorial explains how to set up Graylog for centralized log aggregation using Docker Compose, configure its inputs, and integrate a Spring Boot application with Graylog via Logback GELF, including search syntax examples for effective log analysis.
This article introduces Grafana Loki, an open‑source, horizontally scalable, highly available log aggregation system optimized for Kubernetes and Prometheus, covering its core concepts, architecture, component roles, deployment steps, configuration examples, and practical usage within Grafana.
This guide introduces Loki, the open‑source, horizontally scalable log aggregation system optimized for Prometheus and Kubernetes, covering its core concepts, architecture, components, deployment steps, Grafana integration, label‑based indexing, and best practices for handling dynamic and high‑cardinality tags.
This article explains the motivations for selecting Grafana Loki instead of traditional ELK/EFK stacks, introduces Loki's core concepts and architecture, details component roles, provides step‑by‑step deployment of Promtail and Loki, and demonstrates how to configure and query logs in Grafana while addressing label indexing, dynamic tags, high‑cardinality challenges, and query performance.
This article explains how to build a terabyte‑level log monitoring platform for micro‑service environments by unifying log collection with FileBeat, enriching observability through Elastic APM, processing streams via Kafka Streams, and visualizing metrics with Grafana and Kibana, while addressing cost‑effective filtering and retention strategies.
This guide walks you through setting up a Loki logging cluster for production, covering the native architecture, key configuration differences, storage with boltdb‑shipper on S3, Redis caching, ruler setup, and adapting the Docker‑Compose deployment to Kubernetes.
This article explains the motivations behind choosing Grafana Loki over traditional ELK/EFK stacks for container‑cloud logging, detailing its lightweight design, cost advantages, simple architecture, and how its components—Distributor, Ingester, and Querier—work together to provide scalable, efficient log aggregation and querying.
This article explains Loki's low‑index overhead, concurrent query handling, tag‑based indexing, component roles, read/write paths, step‑by‑step installation of Promtail and Loki, label matching techniques, dynamic‑tag handling, high‑cardinality concerns, and query optimization strategies for cloud‑native log aggregation.
This article explores the challenges of log aggregation in micro‑service architectures, introduces a globally unique log identifier (logid) with its required properties, compares various logid generation schemes, and presents end‑to‑end solutions for log distribution, aggregation, and analysis using custom tools such as ylog and watcher.
This article explains Loki's advantages over Elasticsearch, including low indexing overhead, concurrent query processing with caching, seamless integration with Prometheus and Grafana, detailed architecture components, installation steps, label handling, high‑cardinality challenges, and best practices for efficient log management.
This guide walks you through setting up a Loki logging cluster in production, covering native configurations, extended storage and cache settings, Kubernetes deployment, and practical code examples to simplify the process for newcomers.
Learn step‑by‑step how to design and deploy a billion‑scale log collection and analysis platform using the ELK stack—Filebeat, Kafka, Logstash, Elasticsearch, and Kibana—covering architecture, configuration, installation, and best practices for high‑availability and performance.
This article introduces Loki, Grafana’s lightweight, horizontally scalable log aggregation system, compares it with the EFK stack, explains Promtail, LogQL query language, alerting, and how Loki integrates with Grafana and Prometheus for unified metrics and logs, highlighting its low‑resource, cloud‑native advantages.
This guide walks you through Loki's architecture, advantages, installation via Docker Compose, configuration of Promtail, and step‑by‑step commands to set up a high‑availability, multi‑tenant log aggregation system integrated with Grafana.
This article provides a comprehensive overview of Apache Flume, detailing its purpose as a distributed log aggregation system, explaining its core components such as sources, channels, and sinks, and illustrating its architecture, multi‑agent setups, and key features like reliability, scalability, compression, and monitoring.
This article explains Loki's origins, data model, LogQL query language, low‑cost storage design, and the full read‑write architecture—including Distributor, Ingester, Querier, and QueryFrontend—showing how it solves the shortcomings of traditional Elasticsearch‑based logging solutions and integrates tightly with Prometheus monitoring.
Loki 2.0.0 introduces major enhancements such as a revamped LogQL pipeline, native Prometheus‑style alerts, and simplified storage with boltdb‑shipper, delivering a more resource‑efficient, scalable log aggregation solution for Kubernetes environments.
This article explains the motivations behind choosing Loki over traditional ELK/EFK stacks for container‑cloud logging, outlines its cost‑effective design, describes its simple architecture and components such as Distributor, Ingester, and Querier, and highlights its scalability and seamless integration with Prometheus.
An in‑depth look at Bilibili’s multi‑layer monitoring overhaul, detailing the shift from a monolithic Zabbix setup to micro‑service‑based ELK, Dapper, Misaka, Traceon and Lancer systems, and how layered observability improves fault detection across business, application, and infrastructure levels.
To cope with Meituan Hotel’s exploding micro‑service complexity, the infrastructure team built the Satellite System—combining MTrace and a selective, zero‑intrusion Log4j2‑based logging pipeline that streams enriched logs through Kafka, Storm, Redis and Elasticsearch, delivering second‑level trace‑log queries and six‑month retention, dramatically speeding up debugging.
This article explains the core components of the ELK stack, compares three typical deployment architectures, and provides practical solutions for multiline log merging, timestamp correction, and module‑based log filtering using Filebeat, Logstash, Kafka, and Kibana.
This article introduces CAT, an open‑source Java‑based distributed real‑time monitoring platform, detailing its design goals, architecture, message processing pipeline, logging instrumentation, API, real‑time analysis, report modeling, storage challenges, and key takeaways for building highly available, scalable monitoring solutions.
This article introduces Apache Flume, explains its design goals of reliability, scalability, manageability and extensibility, outlines core concepts and architecture, provides step‑by‑step configuration using the first mode, demonstrates integration with Zookeeper, Kafka and a shell script, and shows how to launch and verify the agent.
This article explains Hadoop 2.0's built‑in MRv2 log collection mechanism, detailing job‑run and task‑run logs, their generation steps, log aggregation, and the role of the JobHistory Server for centralized analysis.
This article explains a practical architecture for aggregating logs from distributed Docker containers by employing Flume NG as a lightweight log collector, Kafka as a high‑throughput message bus, and custom sinks to store logs per service, module and day with low latency and minimal resource impact.
This article provides a comprehensive overview of Apache Flume, covering its design goals, core components, deployment architecture, configuration patterns, and step‑by‑step instructions for integrating Flume with Zookeeper and Kafka to collect and forward massive log data.
This article explains Vipshop’s multi‑layer monitoring architecture, detailing system‑level metrics, application‑level tracing with the Mercury platform, and business‑level KPI dashboards, while describing the data pipelines that collect, process, and alert on distributed logs to ensure reliable operations.
Vipshop’s three‑tier monitoring system—covering system, application (Mercury), and business layers—collects and analyzes logs from distributed components, providing real‑time metrics, slow‑call detection, error tracing, and configurable alerts to help engineers quickly pinpoint and resolve performance issues.
