Learn how to containerize ELK components, configure Rsyslog, deploy Elasticsearch, Logstash, Kibana, and an Nginx logging container, then visualize and query application logs—all with Docker commands and configuration snippets for a complete log‑collection pipeline.
This article examines the challenges of handling massive log data in large‑scale e‑commerce platforms, outlines a baseline ELK‑based architecture, discusses real‑time versus near‑real‑time requirements, and presents four optimization strategies—including basic tuning, platform scaling, data partitioning, and system degradation—to improve performance, resource utilization, and reliability.
This article explores the evolution of operations roles, presents detailed skill maps for system, web, big‑data, and container operations, explains essential log types, and outlines ELK‑based architectures and big‑data‑driven monitoring practices for building a robust, future‑proof operations platform.
This article compiles practical ELK knowledge, covering character‑set conversion, removing unwanted log lines, Grok pattern handling for multi‑line logs, multiline plugin usage in Filebeat and Logstash, date filtering, log type classification, performance optimization, Redis buffering, and Elasticsearch node tuning.
This guide walks through setting up a three‑node ELK stack on Alibaba Cloud, configuring Redis as a broker, installing Filebeat for lightweight log collection, wiring Logstash pipelines, deploying Elasticsearch, Kibana and Grafana, and visualizing Nginx logs with geo‑IP maps and dashboards.
This article walks through the fundamentals of search engines, explaining inverted indexes, the explosion of index size, core Elasticsearch concepts, its distributed architecture, and how it powers the ELK stack for log analysis, all illustrated with clear diagrams and examples.
This article explains why systematic logging is essential for production debugging, introduces a practical log classification and field schema, describes trace‑ID propagation and performance instrumentation, and walks through building an ELK‑based log collection, storage, and real‑time visualization platform for reliable observability.
This tutorial walks you through deploying an ELK stack and a sample web application on Kubernetes, exposing services via NodePort and ClusterIP, scaling the web pods, and using Kibana to visualize logs from multiple containers, all illustrated with YAML files and command‑line examples.
This guide walks through setting up a comprehensive security log collection pipeline—covering WAF, firewall, and Nginx logs—using ELK, Logstash, Kafka, and Flume, and then configuring real‑time alerts with Sentinl or ElastAlert integrated with DingTalk and email notifications.
This article presents a comprehensive guide to monitoring Hadoop Distributed File System (HDFS) on JD Cloud, covering challenges, recommended toolchains, essential metrics, configuration tips, and real‑world case studies to help engineers ensure reliability and performance of large‑scale data clusters.
The article describes the architecture, core functions, and workflow of a MySQL slow‑query analysis system built for the CloudDB platform, covering both daily report generation and real‑time slow‑SQL monitoring using tools such as pt‑query‑digest, ELK, and Kafka.
This article examines the design, deployment, and optimization of massive log systems, comparing architectures, discussing real‑time versus near‑real‑time requirements, and presenting practical improvements such as memory, CPU, network tuning, data partitioning, storage reduction, and component upgrades using ELK, Kafka, Fluentd, and HBase.
This article explains how to monitor Nginx in high‑traffic e‑commerce scenarios, covering essential metrics, latency, error and traffic monitoring, saturation analysis, and visualizing data with ELK and Grafana, plus real‑world case studies and practical configuration tips.
This tutorial walks you through setting up Java, ElasticSearch 2.1.0, Logstash 2.1.1, Kibana 4.3.1, and NGINX on a CentOS 6.7 server, configuring each component, linking them together, and troubleshooting common time‑zone issues so you can visualize logs with Kibana.
This article walks through creating a Python‑based Zhihu crawler, visualizing the harvested user data with the ELK stack, and offers practical improvements such as threading, Redis URL storage, and MongoDB persistence for more efficient and scalable web scraping.
This guide walks through collecting security device and Nginx logs using ELK 5.5.2, Logstash grok patterns, Kafka and Flume pipelines on CentOS 7, and configuring Sentinl or ElastAlert for DingTalk and email alerts, complete with code snippets and deployment commands.
This guide explains how to collect, parse, and visualize MySQL slow query logs using the ELK stack—Filebeat, Logstash, Elasticsearch, and Kibana—covering configuration details, log formatting, and practical querying techniques.
This guide walks you through setting up an ELK stack to collect, forward, and visualize logs from Docker containers, covering Rsyslog configuration, ElasticSearch, Logstash, Kibana, and an Nginx example for end‑to‑end log monitoring.
This guide presents a comprehensive overview of essential ELK Stack utilities—including head plugins, Kibana, ElasticHD, Cerebro, security extensions, visualization platforms, automation frameworks, and alerting solutions—complete with brief feature descriptions and direct links, helping developers and operations teams select the right tools to enhance development, monitoring, and maintenance efficiency.
This article presents a curated collection of ELK Stack utilities—including head plugins, Kibana extensions, cluster monitoring, migration, security, visualization, automation, SQL, and alerting tools—designed to enhance Elasticsearch development, management, and operational efficiency for developers and ops teams.
This article examines the challenges of handling massive log data in high‑traffic e‑commerce platforms and presents a comprehensive architecture, optimization strategies, and practical implementations—including Rsyslog, Kafka, Fluentd, and the ELK stack—to improve scalability, performance, and reliability of log management systems.
This tutorial walks through installing JDK, configuring ElasticSearch, Logstash, Kibana, and Nginx on a CentOS 6.7 server, setting up JSON logging, creating pipelines, and troubleshooting time‑zone issues so you can visualize logs in Kibana.
This article explains how the ELK stack (Elasticsearch, Logstash, Kibana) and Heka can be quickly integrated into mobile games to collect, store, analyze, and alert on log data, improving operational efficiency, troubleshooting speed, and overall service reliability.
This article explains how Elasticsearch and the ELK stack address challenges of storing, securing, retrieving, and analyzing massive data volumes by providing distributed real‑time search, log collection, visualization, and even serving as a NoSQL alternative for large‑scale applications.
This step‑by‑step guide walks you through setting up the ELK stack on CentOS 6, covering Elasticsearch repository configuration, Java installation, Elasticsearch tuning, Logstash pipelines, Kibana deployment, Redis integration, and practical log collection for system, Apache, Nginx, and MySQL logs.
This article introduces the ELK stack—Elasticsearch, Logstash, and Kibana—explaining its core components, architecture, comparison with databases and grep, typical use cases across security, networking, and application monitoring, deployment considerations, challenges, SaaS prospects, and recommended learning resources.
Milano, a distributed log collection and analysis platform built on the ELK stack, leverages Filebeat, Kafka, Logstash, Elasticsearch, and Kibana to provide high‑throughput, low‑latency, secure, and visual log management for massive clusters, addressing the challenges of traditional manual log inspection.
This guide explains what Beats is, its components, typical deployment topology, and provides step‑by‑step instructions for installing, configuring, and visualizing Packetbeat data within an ELK stack, including dashboard setup in Kibana.
This article explores the everyday challenges of operations teams handling system, access, runtime, error, and business logs, outlines the pain points of log collection and standardization, and provides a comprehensive guide to implementing the ELK (Elastic) stack—including Elasticsearch, Logstash, and Kibana—for effective monitoring and analysis.
This article provides a detailed, command‑line walkthrough for installing Java, Elasticsearch, Logstash, and Kibana on Ubuntu, configuring each component with example configuration files, and verifying the ELK stack works by sending test data and querying the REST API.
This article shares a seasoned ops engineer's evolving understanding of IT security, covering network, host, application, data, and web security measures, practical tools like ELK and OSSEC, and actionable recommendations to strengthen system protection against attacks and breaches.
In just over a month, 37 Interactive Entertainment transformed its fragmented monitoring wall into a unified, twelve‑screen dashboard by consolidating game and service data into Elasticsearch, creating a single API, and employing modular JavaScript, custom ECharts visualizations and a 3D map, delivering real‑time insights with a cohesive sci‑fi inspired UI.
This article introduces the ELK stack components, compares four common deployment architectures, and provides step‑by‑step installation and configuration instructions for Elasticsearch, Logstash, Kibana, and Filebeat, including sample command lines and config files.
In this interview, Elastic Stack evangelist Zeng Yong discusses his experiences promoting the ELK stack, the benefits of distributed remote work, ELK's role in big‑data scenarios, comparisons with Hadoop, recent product developments, and upcoming topics for the GIAC conference.
This article explains the core components of the ELK stack, compares three typical deployment architectures, and provides practical solutions for multiline log merging, timestamp correction, and module‑based log filtering using Filebeat, Logstash, Kafka, and Kibana.
This article introduces the ELK stack components, compares three typical deployment architectures—Logstash‑based, Filebeat‑based, and a Kafka‑enhanced version—and provides practical solutions for multiline log merging, timestamp correction, and module‑based filtering using configuration examples.
The article introduces the ELKB stack—a combination of Elasticsearch, Logstash, Kibana, and Beats—explaining its background, user needs, architecture, component functions, typical scenarios, and the team’s practical implementations for real‑time log and time‑series data processing.
This article outlines a large‑scale monitoring solution for distributed microservice environments, comparing traditional ELK logging with a custom GPE stack (Grafana, Prometheus, Exporter, Consul), detailing architecture, components, workflows, and practical considerations for reliable observability.
This article explains why deleted files can still occupy disk space on a Linux server, demonstrates how to identify them with lsof, and shows how configuring Filebeat’s close_older and force_close_files settings can automatically release the space.
This article describes the background, requirements, architecture choices, performance testing, and lessons learned from building a large‑scale, distributed log collection and analysis platform at Hujiang using Elasticsearch, Logstash, Kibana, Filebeat, and Kafka to handle billions of log entries daily.
This article explains how the ELK stack—Elasticsearch, Logstash, and Kibana—provides a flexible, low‑learning‑curve solution for real‑time collection, analysis, and visualization of diverse operational metrics and log data, comparing it with Hadoop, Spark, and traditional databases while highlighting its features, configuration, and best practices.
This guide walks you through installing and configuring the ELK stack (Elasticsearch, Logstash, Kibana), setting up the required environment, creating Logback and Logstash configurations, and integrating them with a Spring MVC application to collect and visualize Java logs in Kibana.
This article outlines the motivation, design principles, architecture, component choices, and step‑by‑step implementation of a comprehensive server security audit system, covering server information collection, log gathering, access control checks, local vulnerability detection, abnormal traffic analysis, and integration with ELK, Hadoop, and open‑source tools like Lynis and OSSEC.
This article walks through creating a Python-based Zhihu web crawler, detailing the tech stack, data collection, visualization of user demographics and top contributors, the crawler architecture, authorization handling, and suggestions for performance and storage improvements.
From early manual deployments to a sophisticated, multi-layered monitoring stack—including ELK, Zabbix, Statsd, Grafana, and Prometheus—Bilibili’s ops team shares the evolution, challenges, and lessons learned in building scalable, automated infrastructure for massive internet traffic.
This guide walks through setting up a distributed log collection and analysis platform using ELK (Elasticsearch, Logstash, Kibana) together with Redis on two CentOS machines, covering environment preparation, component installation, configuration, testing, and visualizing nginx access logs.
Discover how integrating Heka with the ELK stack—Elasticsearch, Logstash, and Kibana—enables mobile game developers to efficiently collect, store, analyze, and visualize logs, offering rapid troubleshooting, automated alerts, and scalable AIOps capabilities that boost operational responsiveness and game performance.
This article explains why developers must adopt mature logging frameworks, use log levels, split logs by time or size, choose appropriate storage backends, and consider unified platforms and real‑time analysis tools like ELK to ensure reliable debugging and monitoring in production.
This article explains how to analyze and visualize Docker container logs using the ELK stack, covering preparation, parsing tips, Kibana query techniques, and example visualizations to help monitor Dockerized environments effectively in production.
This article explains the challenges of Docker log collection and provides a step‑by‑step guide for installing ELK components, configuring Logstash, Kibana, and various log shippers such as Filebeat, logging drivers, Logspout, and Logz.io to reliably gather and visualize container logs.
This tutorial explains how to use Logstash's Grok filter to parse log events, covering basic syntax, custom patterns, built‑in methods like add_field and add_tag, and provides practical configuration examples with visual results.
This article explains how to evolve a Logstash deployment from a minimal single‑instance pipeline to a scalable, fault‑tolerant architecture by adding filters, Filebeat, Elasticsearch clusters, message‑queue buffering, and multi‑connection high‑availability techniques.
Learn step‑by‑step how to install Filebeat, configure it to ship logs to Logstash on an ELK stack, and run the service both directly on the host and via Docker‑Compose, including prerequisite setup, YAML adjustments, and verification through Kibana.
This article explains various enterprise log types, recommends monitoring tools like Cacti, Zabbix, Splunk, and the ELK stack, and details architectures for handling server, application, and user‑click logs using technologies such as Logstash, Elasticsearch, Kibana, Kafka, Flume, and Spark.
This guide explains how to set up the ELK (Elasticsearch, Logstash, Kibana) logging platform using Docker Compose, covering prerequisites, environment configuration, single‑host and multi‑host deployments, and verification steps to get a functional ELK stack in minutes.
Beats are lightweight data shippers in the ELK Stack, and this guide details the purpose, key features, and integration of Filebeat for log collection, Metricbeat for system and service metrics, Packetbeat for network traffic analysis, Winlogbeat for Windows event logs, and Heartbeat for service uptime monitoring.
This guide walks you through downloading, extracting, and starting Elasticsearch, linking it with Logstash, then installing and configuring Kibana, including setting host IP, connecting to Elasticsearch, and accessing the Kibana UI to visualize logs on a Linux server.
The article describes how Qunar designed and evolved its Prism real‑time data platform—leveraging ELK, Kafka, Spark, Docker, and Mesos—to improve data collection, monitoring, and analysis, reduce deployment time, and support scalable DevOps operations across the company.
This article shares practical insights from a senior network manager on why enterprises need Application Performance Management, how to visualize, automate, and intelligent‑ly monitor network infrastructure, define KPI metrics, select tools like Zabbix and ELK, and implement a custom APM platform to improve reliability and reduce operational friction.
Learn how to install and configure the ELK stack (Elasticsearch, Logstash, Kibana, and Filebeat) on Ubuntu 16.04, covering architecture, component roles, download sources, command-line installation, pipeline testing, and integration of Filebeat for real-time log collection and visualization.
This article explores the evolution of log processing in the big‑data era, detailing how ELK Stack, Kafka, and Spark Streaming work together to provide scalable, real‑time log collection, analysis, and visualization for modern cloud‑native operations.
This article describes how Qunar’s Prism platform combines ELK, Kafka, Spark, Docker and other open‑source tools to create a real‑time data pipeline that speeds up problem localization, reduces deployment time, and improves resource utilization across development and operations teams.
This article explains the concept of centralized logging, introduces the ELK stack (Elasticsearch, Logstash, Kibana), compares it with other log solutions, and provides a detailed, cross‑platform installation and configuration guide for Ubuntu 14.04 and CentOS 7.1, including SSL setup and Nginx reverse‑proxy.
This article reviews four ELK deployment patterns, compares their strengths and weaknesses, and demonstrates how ELK can be applied to big‑data operations such as log aggregation, system monitoring, fault diagnosis, and performance analysis with practical examples.
This article recounts how Ctrip’s operations team replaced costly commercial tools with an open‑source ELK stack, detailing requirements, architecture, optimizations, and the impressive scale achieved for real‑time log analysis across thousands of servers.
This guide walks through the complete installation, configuration, and integration of the ELK stack components—Logstash, Redis, Elasticsearch, and Kibana—providing command‑line examples, configuration snippets, testing procedures, and visual verification using Kibana and the Kopf plugin.
This guide introduces Logstash as an open‑source data collection engine, explains its core input‑filter‑output architecture, walks through installation, and demonstrates three practical examples: console I/O, output to Elasticsearch, and reading from Redis with real‑time output.
This guide shows how to compile a modified Bash shell that records each root command with user‑specific identifiers, integrates the logs into an ELK stack, and configures SSH key‑based authentication and server scripts to achieve fine‑grained audit of root activities in Linux environments.
This article explains how internet startups can build a robust logging system by defining log levels, essential log fields, best‑practice logging principles, and choosing between simple file logs or an ELK‑based collection pipeline for monitoring, troubleshooting, and analytics.
This tutorial shows how to quickly set up a log analysis platform using the ELK stack, OpenResty, Ansible, and Vagrant, covering architecture, required tools, server provisioning, deployment commands, testing procedures, and next‑step enhancements for real‑world use.
This guide walks through configuring the firewall, installing Kibana, Logstash, and Elasticsearch, setting up SSL certificates, forwarding ports, deploying Filebeat to client servers, and verifying the ELK stack operation on Linux, complete with command examples and screenshots.
This article walks through the full ELK Stack installation on a Linux test environment, covering component overview, host planning, required packages, detailed command‑line steps for Elasticsearch, Logstash, Kibana and Filebeat, and the necessary firewall and service checks.
The Ctrip Information Security Salon held on August 20 in Shanghai featured expert talks on cloud‑based WAF, big‑data security analytics, ELK‑driven monitoring, product security practices, and concluded with a recruitment drive for security engineers, showcasing practical implementations and industry challenges.
This guide explains what the ELK Stack is, outlines its key advantages, and provides detailed, command-line instructions for installing JDK, Elasticsearch 2.2.0, Kibana 4.4.1, and Logstash on Linux, along with testing, Log4j integration, and a common startup permission fix.
This article details the step‑by‑step evolution of Qunar's domestic flight ticket transaction system, covering early monolithic design, service‑oriented decomposition, database schema migration, adoption of non‑relational storage, and the implementation of an ELK‑based logging and monitoring platform to improve scalability and maintainability.
This guide walks you through centralized log collection, explains the roles of Elasticsearch, Logstash, and Kibana, and provides step‑by‑step commands to install, configure, and verify a complete ELK logging pipeline for Linux servers.
This article describes how to design and implement a closed‑loop data platform using Python, Java, and Spark stacks, covering data acquisition, structuring, mining, visualization, real‑time processing, and deployment with Docker, ELK, Kafka, and cloud services, illustrated by three industry case studies.
This article shares the author's experience and lessons learned while building a log‑monitoring framework with the ELK stack, discussing performance issues, configuration of Logstash filters using Grok, and practical tips for deploying ElasticSearch, Logstash, and Kibana in production environments.
This article explains why log analysis platforms are essential as systems grow, outlines the benefits of centralized logging, presents Ctrip’s real‑world requirements and challenges, and introduces the ELK stack as a scalable solution for collecting, storing, and visualizing massive log data.
This guide walks you through the fundamentals of big‑data characteristics, the challenges of log management for small‑scale operations, and provides a detailed, step‑by‑step tutorial for installing and configuring the ELK stack (Elasticsearch, Logstash, Kibana) on a CentOS 6.6 server, including all required dependencies.
The Meizu Tech Open Day showcased the company's rapid evolution to a data‑driven mobile internet firm, detailing its DW1.0 and DW2.0 data‑warehouse architectures, recommendation pipelines, Spark adoption, and ELK‑based log analytics, while sharing practical lessons and future challenges.
This article explores the evolution of cloud platforms—from early virtualization (Cloud1.0) through resource‑centric Cloud2.0 to application‑centric PaaS (Cloud3.0)—detailing essential operations tasks such as resource allocation, deployment, service discovery, and log management, and outlining practical design choices using Docker, Mesos, and ELK.
This article shares the author's practical experience building a real‑time log analysis platform at Sina, covering service scope, ELK architecture, performance optimizations, usability improvements, new features, common pitfalls, and a concise Q&A for engineers handling massive log streams.
This article explores how logs serve as vital diagnostic tools in operations, detailing their definition, practical uses for performance and security monitoring, step‑by‑step analysis techniques, and recommendations for free and commercial log analysis solutions.
The fourth ESCC domestic developer conference held at Ctrip's headquarters in Shanghai gathered around 100 engineers to share ElasticSearch product roadmaps, core search system experiences, Kibana 4 features, and best practices for building high‑throughput ELK clusters.
This article describes how Qunar's DevOps team built a Mesos‑powered log analysis platform using ELK, Marathon, Chronos, and custom monitoring frameworks, detailing deployment architecture, resource management, container logging, image handling, and operational lessons learned.
The article shares Meiya's journey from a small startup to a full‑featured beauty video platform, detailing common early‑stage challenges, communication and protocol strategies, RESTful API design, development workflow, toolchain choices like Laravel, DB migrations, iOS libraries, ELK logging, and DevOps automation with SaltStack.
This article compiles the Q&A from a high‑level operations talk where experts discuss ELK cluster sizing, performance tuning, schema handling, integration with Hadoop and Splunk, and practical tips for managing massive log streams in production environments.
This article explores how the ELK stack can transform large‑scale log processing into fast, flexible, and interactive analysis for troubleshooting, security auditing, and monitoring, sharing practical examples, common pitfalls, and best‑practice recommendations from real‑world deployments at Sina.
This guide explains why log analysis is critical, outlines common challenges, compares heavyweight trace systems with the lightweight LEK stack, and provides step‑by‑step installation, configuration, and best‑practice tips for Logstash, Elasticsearch, and Kibana.
