This guide shows how to create a Bash script that compresses and backs up yesterday's log files, removes the originals, and schedules the process with a daily cron job to prevent disk‑space exhaustion on Linux systems.
This article explains what Elasticsearch is, how it solves database acceleration, log observability, and data analysis problems, details its core components and underlying engine features, compares its strengths and weaknesses, and presents classic application scenarios and a real‑world case study integrating Elasticsearch with Flink for large‑scale log analytics.
Vipshop’s Dragonfly log platform evolved from a costly 260‑node Elasticsearch cluster to a ClickHouse‑based architecture that uses a unified JSON format, vfilebeat ingestion, Flink parsing, and MergeTree storage to achieve high‑throughput writes, fast vectorized queries, flexible TTL management, and dramatically lower operational expenses.
This article introduces the ELK stack as a popular centralized logging solution, explains three typical deployment architectures—including Logstash‑based, Filebeat‑based, and cache‑queue‑enhanced setups—details their advantages and drawbacks, and provides practical configuration examples and troubleshooting techniques for multiline logs, timestamp handling, and module filtering.
This article introduces the ELK stack components, compares three typical deployment architectures—Logstash as collector, Filebeat as collector, and a cache‑queue‑enhanced design—then discusses common logging issues such as multiline merging, timestamp handling, and module filtering, providing concrete configuration examples and solutions.
This article explains the challenges of traditional open‑source log collection in cloud‑native environments, describes Volcano Engine’s unified TLS architecture, its centralized configuration, CRD‑based deployment, and showcases real‑world case studies that demonstrate improved availability, efficiency, and scalability.
This article introduces the ELK stack components, explains why centralized logging is essential, outlines core log‑system features, compares three ELK architectures, provides step‑by‑step installation and configuration for Filebeat, Logstash, Elasticsearch and Kibana, and demonstrates a complete pipeline using Kafka with code examples and diagrams.
This article introduces the ELK stack components, compares three typical deployment architectures—including Logstash‑based, Filebeat‑based, and Kafka‑enhanced setups—and provides detailed configuration examples and solutions for multiline log merging, timestamp handling, and module‑specific filtering.
This article explains how to use Filebeat and Graylog together for centralized log collection, covering Filebeat’s role, configuration files, input modules, Graylog’s architecture, pipeline rules, and step‑by‑step deployment using Docker and docker‑compose, providing practical commands and examples for operational environments.
This article recounts a year‑long journey of designing, implementing, and operating a multi‑environment Kubernetes architecture—including containerized workloads, unified logging, CI/CD pipelines, service governance with Istio, and private deployments—while sharing practical lessons and best‑practice recommendations for cloud‑native teams.
This article explains why centralized log collection is essential for multi‑environment services, introduces Graylog’s architecture, details Filebeat’s role and workflow, provides configuration examples, and walks through Docker‑based deployment of both Filebeat and Graylog for robust log management.
This guide explains why Loki is a lightweight alternative to EFK/ELK, walks through installing Loki and Promtail binaries, configuring them with YAML files, and visualizing logs in Grafana using LogQL, providing a complete end‑to‑end log management solution.
This document details the motivation, architecture, components, query language, and deployment of a Loki‑based log collection and analysis platform for JD Daojia, comparing it with ELK, describing ingestion, real‑time and historical log handling, technical challenges, configuration examples, and future scaling plans.
This guide explains how to use the built‑in Logrotate tool on Linux to configure automatic log rotation, customize rotation policies for services like Nginx, PHP and syslog, troubleshoot common issues, and replace ad‑hoc scripts with reliable, cron‑driven solutions.
Version 2.3.0 of Loki introduces enhanced features such as per‑tenant custom retention policies, time‑range log deletion via the Compactor API, Prometheus‑style recording rules, a new pattern parser for LogQL, ingestion sharding for faster queries, and advanced IP‑matching syntax, all aimed at improving storage efficiency, compliance, and observability.
This tutorial explains how to set up a real‑time ELK log analysis platform, covering the architecture of Elasticsearch, Logstash and Kibana, detailed installation commands, configuration for Spring Boot and Nginx logs, and how to run the stack continuously with Supervisor.
This article outlines how to design, print, collect, and manage online service logs efficiently—covering log levels, key information, formatting, rolling, local vs. remote storage, real‑time collection, and tool selection—to turn logs into a valuable debugging and analytics asset.
This article explains how Loki, a lightweight, Prometheus‑compatible logging system, addresses the high resource cost, complexity, and operational overhead of traditional ELK/EFK stacks by using label‑based indexing, efficient compression, and scalable architecture for container‑cloud environments.
This article provides a detailed overview of the ELK stack—including Elasticsearch, Logstash, Kibana, and Beats—explaining its components, why to use it for centralized log management, various deployment architectures, system tuning, security setup, and step‑by‑step installation and configuration commands for a production‑grade environment.
This article explains what log auditing is, why a unified audit platform is essential for compliance and security operations, outlines its core goals, main functions such as unified collection, correlation analysis, real‑time alerts, forensic reporting, and describes typical deployment architectures and modules.
The Xianyu Android Log Governance solution unifies local console logs, TLog and online logs, adds comprehensive logcat capture and AUS‑OSS retrieval, replaces BlockCanary with low‑overhead frame‑callback lag/ANR detection, and provides dashboards and a batch‑query platform, cutting technical‑opinion ratio from 10.5 % to 4.7 % and boosting upload success.
Obsuite, DiDi’s open‑source observability suite, tackles hybrid‑cloud monitoring challenges by combining metrics, logs, and traces, while the article analyzes market trends, private‑cloud demand, and the product’s architecture, open‑source components, and the OCE certification program for enterprise users.
This article provides a comprehensive introduction to the ELK stack—Elasticsearch, Logstash, Kibana, and Filebeat—including its components, why it’s used for centralized log management, detailed architecture diagrams, step‑by‑step installation commands, configuration examples, and a practical Kafka‑based data pipeline demonstration.
During a high‑traffic promotion a service ran out of disk space because massive log files weren't cleaned, the investigation revealed a lingering SLS process holding deleted files, and the article walks through the root‑cause analysis, the kill‑process fix, and a Spring‑Boot starter that dynamically degrades log levels to prevent recurrence.
This article explains what MySQL's slow query log is, how to configure and read it, and provides step‑by‑step procedures for safely clearing the log in both FILE and TABLE output modes to free disk space and maintain database performance.
Youzan’s log‑governance guide uses a car‑dashboard analogy to show why precise ERROR logs and sensible alerts matter, defines INFO/WARN/ERROR levels, sets daily reduction targets, leverages top‑error analysis and water‑level monitoring, and ultimately cut daily ERROR entries from thousands to about one hundred while catching issues before incidents.
This article details how Dada’s SRE team rebuilt its logging platform—replacing ELK with Filebeat and Storm, deploying Elasticsearch hot and cold nodes on Kubernetes, and optimizing log collection, parsing, and storage to handle over 130 billion daily log entries and 300 TB of data.
During a large‑scale promotion, an online service suffered severe disk usage spikes due to undeleted log files held open by an SLS process, and the issue was resolved by identifying the open handles, terminating the process, and implementing log‑level controls to prevent recurrence.
This article provides an in‑depth tutorial on using the Linux logrotate utility, covering default configuration files, command‑line options, example configurations for services like Nginx, PHP‑FPM, Tomcat, custom Bash and Python rotation scripts, troubleshooting tips, and how to adjust cron scheduling for automated log rotation.
This article examines the challenges of log management in Kubernetes environments and compares five popular solutions—Zebrium, Sematext, Loki, ELK Stack, and Fluentd—highlighting their key features, advantages, and limitations to help you choose the right tool.
This guide explains how to locate Docker container log files on Linux, provides shell scripts to list and truncate oversized logs, and shows how to configure per‑container and global log size limits using docker‑compose and the Docker daemon to keep host disk usage under control.
This article provides a comprehensive walkthrough of Logrotate on Linux, covering its purpose, default configuration files, key directives, practical examples for Nginx, PHP, Tomcat, and system logs, as well as custom shell and Python scripts for advanced log rotation and troubleshooting.
This article provides a thorough walkthrough of Linux logrotate, covering its default configuration files, essential command‑line options, key parameters, practical examples for services like nginx, php, and tomcat, custom rotation scripts, cron scheduling, and troubleshooting tips for reliable log management.
This guide explains how to use Logrotate on Linux for automatic log file rotation, covering default configuration files, key options, example setups for Nginx, PHP, Tomcat, system logs, custom shell and Python scripts, cron scheduling, and troubleshooting techniques to ensure logs are rotated reliably.
This article walks through a mid‑size gaming company's journey of deploying, tuning, and scaling an Elasticsearch cluster for massive log volumes, covering hot‑cold node architecture, ILM policies, shard management, Logstash‑Kafka optimization, emergency expansions, and the promise of searchable snapshots to achieve petabyte‑scale storage with cost efficiency.
This article details how Dada built a Kubernetes‑mixed log platform that handles over 130 billion logs per day, stores more than 14 TB daily, and maintains a 300 TB total volume by automating collection with Filebeat, parsing with Storm, and optimizing Elasticsearch with hot‑cold nodes.
This article provides an in‑depth overview of Apache Kafka’s internal topics, preferred replicas, partition allocation mechanisms, log directory structure, index files, offset and timestamp lookup, log retention and compaction policies, storage architecture, delayed operations, controller role, consumer rebalance process, and producer idempotence.
The article describes how a distributed deployment platform built a centralized Elasticsearch‑based log collection and query system to replace manual multi‑machine log inspection, detailing the background challenges, architecture, implementation steps, practical usage, and future improvements.
The article outlines seven essential performance metrics—response time and throughput, average load, error rate, GC pause time, business indicators, uptime, and log size—explaining their significance for application health and recommending popular monitoring tools for each metric.
This step‑by‑step guide shows how to diagnose exploding log files, configure logrotate with size‑based and time‑based rules, use cron jobs for frequent rotations, and apply options like copytruncate, delaycompress, and dateext to keep production systems healthy and storage under control.
This guide walks you through configuring Kibana visualizations, customizing Nginx and Apache logs in JSON, setting up Logstash pipelines, integrating Redis as a buffering queue, and deploying a production-ready ELK stack for efficient log management and analysis.
This comprehensive tutorial walks you through installing Java, Elasticsearch, Logstash, Kibana, and related tools on two CentOS 7 servers, configuring cluster settings, verifying health, and visualizing logs with Kibana, complete with command‑line examples and troubleshooting tips.
This article explains the motivations for moving a high‑volume operation‑log system from MongoDB to Elasticsearch, outlines the existing architecture, details capacity planning, index design, and a step‑by‑step migration process using Kafka, DataX, and Spring Boot, and shares the performance gains and lessons learned.
The article explains the design and implementation of Youzan’s ZanLogger, a mobile logging platform that streamlines issue diagnosis by providing secure, efficient log collection, retrieval, and analysis features—including active and passive reporting, mmap‑based I/O reduction, dynamic symmetric key encryption, and multi‑dimensional storage—while detailing performance benchmarks and future enhancements.
This guide explains how to combine ELK (Elasticsearch, Logstash, Kibana) with Zabbix using the logstash-output-zabbix plugin, configure Logstash pipelines to filter error keywords, and set up Zabbix templates and triggers for instant log‑based alerts.
This guide shows how to extend the open‑source ELK stack by installing the logstash‑output‑zabbix plugin, configuring Filebeat and Logstash to filter network device logs, and linking them to Zabbix for automated real‑time alerts, complete with sample configurations and verification steps.
This guide walks through deploying a centralized Rsyslog server on CentOS, configuring SELinux, firewall, rsyslog, Filebeat, Logstash, and Kibana to collect, process, and visualize logs from various network devices, addressing common sysadmin challenges.
The article outlines how a company built a centralized cron‑job platform—using Python’s crontab library, SaltStack deployment, ELK log aggregation, and automated email alerts—to integrate existing tasks, provide reliable CRUD operations, enable fast log querying, and detect failures, cutting operational overhead while managing thousands of scheduled jobs across multiple servers.
This article describes how 58 Group's DB department evaluated and adopted Elastic Stack to build a scalable, reliable centralized log processing system, detailing its components, four architectural patterns, and a MySQL audit log example for unified log collection, storage, analysis, and alerting.
This article provides a comprehensive, step‑by‑step tutorial on using Linux command‑line tools such as cat, tail, vim, grep, sed, ps, netstat, lsof, and free to efficiently view large log files, locate specific entries, monitor processes and ports, and assess overall system health.
This article provides a detailed tutorial on designing, deploying, and operating an ELK log management platform—including Elasticsearch, Logstash, Kibana, Kafka, and Filebeat—covering architecture options, configuration files, command‑line operations, cluster setup, and best‑practice recommendations for scalable, real‑time log collection and analysis.
This article examines the challenges of handling massive log data in large‑scale e‑commerce platforms, outlines a baseline ELK‑based architecture, discusses real‑time versus near‑real‑time requirements, and presents four optimization strategies—including basic tuning, platform scaling, data partitioning, and system degradation—to improve performance, resource utilization, and reliability.
This tutorial walks you through installing Graylog with Docker‑Compose, configuring it for Spring Boot, and routing Logback and Log4j2 logs to Graylog, while also comparing Graylog to the ELK stack and providing source links for the full projects.
This article explores the evolution of operations roles, presents detailed skill maps for system, web, big‑data, and container operations, explains essential log types, and outlines ELK‑based architectures and big‑data‑driven monitoring practices for building a robust, future‑proof operations platform.
This article details Youzan's evolution from a simple Flume‑based log collector to a multi‑tenant, Kafka‑buffered, Spark‑processed, HBase‑backed logging architecture that now handles hundreds of billions of log entries per day, highlighting challenges, design decisions, and future improvements.
This article examines the design, deployment, and optimization of massive log systems, comparing architectures, discussing real‑time versus near‑real‑time requirements, and presenting practical improvements such as memory, CPU, network tuning, data partitioning, storage reduction, and component upgrades using ELK, Kafka, Fluentd, and HBase.
This article explores how AI-driven AIOps can turn massive operational log data into actionable insights, detailing the five‑level AI capability model, real‑world implementation scenarios, and industry case studies that demonstrate the value of an intelligent log center.
This guide shows how to automatically split growing Linux log files into 60 MB chunks with date‑based naming, using the `date` and `split` commands in a Bash script, and then safely truncate the original log to free disk space.
This article describes how a rapidly growing technical team built a unified log system that consolidates program, web access, and slow logs, introduces host‑agent and process‑agent collection, leverages Kafka, Elasticsearch, and Storm for high‑throughput processing, and provides monitoring, alerting, and reporting features to improve reliability and operational efficiency.
Amid the rise of serverless Kubernetes, growing pod volumes, and real-time log demands, this article examines emerging log-handling challenges, evaluates traditional collection methods, and presents a comprehensive “Logtail + Log Service + Ecosystem” architecture that delivers high-throughput, reliable, and scalable logging for cloud-native environments.
This article examines the challenges of handling massive log data in high‑traffic e‑commerce platforms and presents a comprehensive architecture, optimization strategies, and practical implementations—including Rsyslog, Kafka, Fluentd, and the ELK stack—to improve scalability, performance, and reliability of log management systems.
This article explains why log testing is essential for server‑side applications, describes four log categories, provides concrete configuration examples, and lists eight practical checkpoints to ensure logs are useful, secure, and reliable during development and production.
This article introduces the ELK stack—Elasticsearch, Logstash, and Kibana—explaining its core components, architecture, comparison with databases and grep, typical use cases across security, networking, and application monitoring, deployment considerations, challenges, SaaS prospects, and recommended learning resources.
Logan is Meituan‑Dianping’s native C‑based mobile logging library that uses MMAP caching, streaming compression and encryption, and a unified protocol to eliminate Java‑heap GC pauses, prevent log loss, secure data, and provide efficient retrieval and proactive reporting across dozens of apps.
In this interview, Elastic Stack evangelist Zeng Yong discusses his experiences promoting the ELK stack, the benefits of distributed remote work, ELK's role in big‑data scenarios, comparisons with Hadoop, recent product developments, and upcoming topics for the GIAC conference.
This guide details practical solutions for TDH Community Edition operational problems, covering component configuration paths, log locations, disk‑space alerts, service start/stop via kubectl, time synchronization, network/firewall checks, hostname resolution, and license service management.
This article details Bilibili's Billions log platform—from its fragmented origins and design goals to the elastic‑stack‑based architecture, shard management, log sampling, custom Go splitters, and monitoring enhancements—highlighting the challenges faced and the roadmap for future improvements.
This article describes the background, requirements, architecture choices, performance testing, and lessons learned from building a large‑scale, distributed log collection and analysis platform at Hujiang using Elasticsearch, Logstash, Kibana, Filebeat, and Kafka to handle billions of log entries daily.
This guide explains why removing Apache/Tomcat log files while the services are running fails to release disk space, analyzes the inode‑locking behavior, and provides practical solutions such as restarting the server, truncating logs, and using scheduled cleanup commands.
This guide explains why logrotate is essential for Linux system security, details its default configuration files, shows how the cron‑based rotation works, lists key options, provides practical Nginx and custom rotation examples, and highlights the critical copytruncate option for seamless log handling.
This article explains how building complete, accurate, and real‑time foundational data—covering CMDB, logs, databases, and knowledge bases—forms the essential base for reliable IT operations, detailing the three core attributes, practical collection methods, and the structure of each data module.
Discover how integrating Heka with the ELK stack—Elasticsearch, Logstash, and Kibana—enables mobile game developers to efficiently collect, store, analyze, and visualize logs, offering rapid troubleshooting, automated alerts, and scalable AIOps capabilities that boost operational responsiveness and game performance.
This article describes the architecture, deployment, configuration, maintenance, and performance results of a large‑scale near real‑time logging platform built with Kafka, Flume, and Elasticsearch, highlighting practical lessons and future plans for resource‑efficient operation.
This article explains the purpose and generation of Tomcat's default log files—catalina.out, catalina.{yyyy‑MM‑dd}.log, and localhost.{yyyy‑MM‑dd}.log—how they are configured via logging.properties and server.xml, and which types of messages appear in each file to aid troubleshooting.
This article explains why developers must adopt mature logging frameworks, use log levels, split logs by time or size, choose appropriate storage backends, and consider unified platforms and real‑time analysis tools like ELK to ensure reliable debugging and monitoring in production.
This article explains how to evolve a Logstash deployment from a minimal single‑instance pipeline to a scalable, fault‑tolerant architecture by adding filters, Filebeat, Elasticsearch clusters, message‑queue buffering, and multi‑connection high‑availability techniques.
Learn step‑by‑step how to install Filebeat, configure it to ship logs to Logstash on an ELK stack, and run the service both directly on the host and via Docker‑Compose, including prerequisite setup, YAML adjustments, and verification through Kibana.
This guide explains how to set up the ELK (Elasticsearch, Logstash, Kibana) logging platform using Docker Compose, covering prerequisites, environment configuration, single‑host and multi‑host deployments, and verification steps to get a functional ELK stack in minutes.
This article explains the concept of centralized logging, introduces the ELK stack (Elasticsearch, Logstash, Kibana), compares it with other log solutions, and provides a detailed, cross‑platform installation and configuration guide for Ubuntu 14.04 and CentOS 7.1, including SSL setup and Nginx reverse‑proxy.
The article explains the design, implementation, and operational benefits of a middleware component that dynamically adjusts log levels in Java services, helping large‑scale delivery platforms balance comprehensive debugging information with system performance and avoid catastrophic failures during peak traffic.
This article explains how internet startups can build a robust logging system by defining log levels, essential log fields, best‑practice logging principles, and choosing between simple file logs or an ELK‑based collection pipeline for monitoring, troubleshooting, and analytics.
This guide walks you through centralized log collection, explains the roles of Elasticsearch, Logstash, and Kibana, and provides step‑by‑step commands to install, configure, and verify a complete ELK logging pipeline for Linux servers.
The article gathers expert insights on building an integrated monitoring system for the securities industry, covering common pitfalls, the need for standardization, architectural design principles, KPI definitions, trend analysis techniques, and practical tool recommendations for effective operations.
This article explains how Kafka's LogManager loads existing logs, manages background threads for flushing, checkpointing, cleaning, and compaction, and details the code implementations and strategies for log retention, segment cleanup, and log compression in a distributed storage environment.
This article introduces Splunk as a specialized IT search engine for big data, detailing its data collection, indexing, analysis, reporting, security integration, scalability, and practical use cases such as operational troubleshooting, compliance auditing, and application performance monitoring.
This guide walks you through the fundamentals of big‑data characteristics, the challenges of log management for small‑scale operations, and provides a detailed, step‑by‑step tutorial for installing and configuring the ELK stack (Elasticsearch, Logstash, Kibana) on a CentOS 6.6 server, including all required dependencies.
An in‑depth guide walks readers through building a comprehensive monitoring system for small‑to‑medium enterprises, covering hardware, system, application, network, security, traffic analysis, business metrics, log aggregation, automation, visualization, and practical integration with tools like Zabbix, IPMI, ELK, and Smokeping.
This guide explains how logrotate automatically truncates, compresses, and removes old log files, covering installation on major Linux distributions, essential configuration options, practical examples, and troubleshooting techniques to keep server storage under control.
Learn essential log analysis techniques using command‑line tools like grep, cut, and awk, explore advanced filtering with rsyslog and Grok, and discover how log management systems can automate parsing and accelerate troubleshooting of Linux syslog data.
This article presents a comprehensive set of logging best‑practice rules—from defining log levels and classifications to using RequestIDs, monitoring alerts, and managing log size—aimed at improving system reliability, troubleshooting speed, and operational efficiency.
This article compiles the Q&A from a high‑level operations talk where experts discuss ELK cluster sizing, performance tuning, schema handling, integration with Hadoop and Splunk, and practical tips for managing massive log streams in production environments.
This guide explains how to use logrotate on Linux to automatically rotate, compress, and delete log files, covering installation, configuration examples for size‑based and time‑based rotation, and essential options to keep server storage under control.
