This article explains why front‑end error logging is essential, how BadJS (JavaScript runtime errors) are captured, enriched, and reported, and provides practical strategies for analyzing logs, handling Script errors, dealing with hybrid WebView environments, and scaling the system with big‑data pipelines.
This comprehensive guide explains why monitoring is essential for operations, outlines monitoring goals and methods, reviews a wide range of open‑source tools, details a Zabbix‑based workflow, enumerates key metrics across hardware, system, application, network, security and business layers, and offers practical alerting and interview tips.
The article explains why loading all log data into a database is impractical, outlines three drawbacks—volatile requests, data bloat, and cost—and introduces the lightweight awk tool with concrete command examples to filter and analyze network logs efficiently without a database.
This guide shows how to generate hourly slow‑query reports from MySQL slow‑log files by using pt‑query‑digest’s timeline feature, filtering with sed, importing the data into SQLite with termsql, and then querying the results to identify hotspot periods and recurring patterns.
This tutorial walks you through installing the GD library, building SARG from source, configuring its key parameters, running reports, using command‑line options, scheduling automated jobs with cron, and troubleshooting common installation and configuration issues for Squid log analysis.
This article presents a modular architecture for real‑time ETL log collection and analysis, detailing the installation and configuration of Flume, Kafka, Storm, Drools, and Redis, and explains how their integration improves fault tolerance, scalability, and processing speed.
When log files grow massive, traditional tools like vim, cat, grep, and awk become slow and memory‑hungry, but Linux’s split command lets you divide a huge file by line count or size, process the pieces individually, and later recombine them, dramatically improving analysis efficiency.
In an interview, Tencent Cloud senior engineer Chen Xi explains how Tencent optimizes Elasticsearch for massive log, monitoring, and document search workloads by prioritizing stability through kernel tweaks, boosting performance with scenario‑specific settings, cutting costs via redundant storage trimming and off‑heap indexing, and leveraging rich data‑pipeline components and robust distributed cluster management to lower operational overhead while anticipating future ecosystem growth and community collaboration.
This guide explains why web log analysis is essential for security, demonstrates how to parse Apache logs, distinguishes normal from malicious requests, and provides practical Secsoso commands for business behavior statistics, traffic monitoring, and detecting attacks such as CC, SQL injection, file inclusion, and XSS.
Monitoring network activity and ensuring compliance requires effective log analysis, and this article reviews five open‑source tools—Graylog, Nagios, Elastic Stack, LOGalyze, and Fluentd—detailing their features, strengths, and use cases for operations and security teams.
This article details Ele.me Logistics' mobile‑app monitoring architecture—E‑Monitor, TimeBomb, Dogger, and EDW—explaining how each layer collects, visualizes, and analyzes business‑level availability data, and showcases a real‑world debugging case that leveraged the stack to resolve an HTTP/2 connectivity bug.
The April 27 Elastic Nanjing Technical Exchange, co‑hosted by Suning Technology and the Elastic Chinese community, showcased how Suning leverages Elasticsearch, Logstash, Kibana and Beats for real‑time search, log analysis, security, and smart‑retail applications, featuring expert case studies and future big‑data strategies.
This article explains how to write two Bash scripts that parse web server access logs with awk and date to automatically list IP addresses that caused failed requests and to list IPs that accessed the site within the last ten minutes.
Pfinder is a new tool that combines problem finder and performance finder functionalities to help users visualize call relationships and time consumption distributions for efficient problem localization in distributed systems.
This guide outlines ten practical methods for Linux administrators to identify compromised machines on CentOS, including log inspection, checking /etc/passwd and /etc/shadow, analyzing login records, monitoring network traffic, and using lsof to recover deleted log files, with step‑by‑step commands and examples.
This article explains the fundamentals of Kubernetes audit logs, their JSON format, recording stages and levels, and shows how to configure policies, analyze logs, and use Alibaba Cloud's integrated solution to create visual reports, custom alerts, and advanced queries for security monitoring.
This guide compiles a comprehensive set of Linux command‑line techniques for parsing Apache and other web server logs, enabling you to count unique IPs, identify hot pages, filter bots, measure bandwidth, track connection states, and spot performance bottlenecks in a single, actionable reference.
This article explains what Android Monkey testing is, walks through the preparation steps, command usage, log redirection, and basic Python-based log filtering, and suggests how to automate the process to improve mobile app stability before release.
This guide compiles a comprehensive set of Linux commands—using awk, grep, netstat, and more—to help you analyze web server logs, track traffic, identify top IPs, monitor connection states, and detect performance bottlenecks on an Alibaba Cloud ECS instance.
This guide walks Linux administrators through common signs of server compromise, shows how to examine logs, user files, active processes, network traffic, and demonstrates using lsof and /proc to recover deleted log files, all with concrete command examples.
The article explains why a downtime diagnosis system is needed, outlines its architecture and implementation methods—including log sources, feature extraction, and API integration—and presents early results showing high automation coverage and significant operational cost savings.
The article explains what a big data development engineer does, the tools and skills required such as Hadoop, Hive, Spark and Kafka, how they process massive logs to compute metrics like PV and UV, and compares this role with conventional business system development.
This guide walks Linux operations engineers through common signs of a breached machine, showing how to examine logs, verify user files, monitor login events, use lsof for deleted files, and recover critical logs with step‑by‑step commands.
This guide explains how to set up a Python‑based log‑analysis tool for web servers, defines key terminology, outlines installation requirements, describes its abstract‑based aggregation features, and demonstrates usage of the accompanying command‑line utility for request, IP, distribution, and detail analysis.
This article details a real-world incident response to a compromised website that redirected visitors to a gambling site, covering intrusion analysis, server remediation, log tracing, and comprehensive cleanup steps to restore security.
This article explains how Elasticsearch and the ELK stack address challenges of storing, securing, retrieving, and analyzing massive data volumes by providing distributed real‑time search, log collection, visualization, and even serving as a NoSQL alternative for large‑scale applications.
Milano, a distributed log collection and analysis platform built on the ELK stack, leverages Filebeat, Kafka, Logstash, Elasticsearch, and Kibana to provide high‑throughput, low‑latency, secure, and visual log management for massive clusters, addressing the challenges of traditional manual log inspection.
This article introduces a Python‑based log analysis tool for web servers that provides minute‑level aggregation, abstracted URI and argument patterns, and multi‑dimensional performance metrics, along with installation steps, core features, implementation details, usage commands, and deployment guidelines.
This article reviews the PreFix system, which uses machine‑learning on datacenter switch logs to predict hardware failures ahead of time, detailing its design, feature extraction, random‑forest model, experimental validation across multiple switch models, and its broader applicability to disk failure prediction.
This guide walks through eleven hands‑on techniques for uncovering UNIX or Solaris intrusions, from inspecting password files and processes to verifying daemon configurations, network sockets, logs, core dumps, hidden files, file integrity, kernel modules, and the limits of manual detection versus IDS solutions.
This article explains how to create a lightweight yet reliable Python‑based log analysis tool that parses nginx logs with regular expressions, stores detailed metrics in MySQL, and provides fine‑grained performance and anomaly reports for web services.
This article explores applying big‑data thinking and platforms—such as Flume, Spark Streaming, and HBase—to operations monitoring, detailing data sources, metric categories, architecture design, implementation steps, and the benefits of a scalable, low‑code monitoring platform.
To mitigate startup latency spikes in Java-based query services caused by class loading, JIT warm‑up, and lazy resource loading, the article presents a generic, low‑cost pre‑warming component that parses local Dubbo and HTTP logs, filters, samples, and replays traffic, detailing its design, implementation, and performance optimizations.
This article outlines the motivation, design principles, architecture, component choices, and step‑by‑step implementation of a comprehensive server security audit system, covering server information collection, log gathering, access control checks, local vulnerability detection, abnormal traffic analysis, and integration with ELK, Hadoop, and open‑source tools like Lynis and OSSEC.
This article presents five practical Python scripts that demonstrate how to traverse directories and list files, draw text triangles, create a number‑guessing game, log disk usage, and analyze IP access counts from log files, each explained with clear code examples.
This article explains how to analyze and visualize Docker container logs using the ELK stack, covering preparation, parsing tips, Kibana query techniques, and example visualizations to help monitor Dockerized environments effectively in production.
This article explains Google’s BeyondCorp concept, the need for deep defense of internal and perimeter networks, and provides practical Linux scripts for monitoring processes, ports, command usage, system events, file changes, and SFTP activity to detect and mitigate host intrusions.
Deep DefenseLinux monitoringhost intrusion detection
Alibaba’s SunFire platform delivers massive‑scale, real‑time log collection, processing, and visualization for e‑commerce spikes like Double 11, using low‑overhead agents, asynchronous Map/Reduce pipelines, fault‑tolerant task scheduling, and shared inputs to ensure accurate, low‑latency monitoring across billions of transactions.
The article critiques popular but unnecessary log‑analysis features—such as sub‑second alerts, endless pagination, flashy maps, full SQL support, bulk downloads, and live tail—arguing that focusing on practical alert content, efficient querying, and proper architecture yields far more value for IT operations.
This article describes Ctrip's challenges with web security, evaluates hardware and commercial cloud WAF shortcomings, and presents a low‑cost, low‑risk cloud‑based WAF solution that leverages DNS redirection, closed‑loop rule management, Lua/Tengine deployment, supervised machine‑learning log analysis, and big‑data streaming for real‑time threat detection and mitigation.
This tutorial shows how to quickly set up a log analysis platform using the ELK stack, OpenResty, Ansible, and Vagrant, covering architecture, required tools, server provisioning, deployment commands, testing procedures, and next‑step enhancements for real‑world use.
This article explains how to extract valuable performance and traffic insights from Nginx access logs using shell commands and awk, covering request volume, peak rates, bandwidth usage, slow‑query detection, URL normalization, and practical optimization recommendations for web operations.
This article explains how to apply big‑data concepts and platforms to operations monitoring, covering data sources, metric extraction from logs, architectural design with Flume, Spark Streaming and HBase, implementation steps, and the resulting benefits for scalability and rapid metric development.
This guide explains essential Linux audit commands, log‑viewing techniques, user and process inspection methods, and additional checks such as file integrity, network monitoring, cron jobs, and backdoor detection, providing a comprehensive toolkit for system security and operations.
This article summarizes a 2016 Global Operations conference talk that explains the concept of IT Operations Analytics (ITOA), its four data sources, the evolution of log management from databases to real‑time search engines, and real‑world case studies demonstrating how fast, large‑scale log analysis improves monitoring, security, and business insight.
This article explains why log analysis platforms are essential as systems grow, outlines the benefits of centralized logging, presents Ctrip’s real‑world requirements and challenges, and introduces the ELK stack as a scalable solution for collecting, storing, and visualizing massive log data.
This guide explains how to use the lightweight ngxtop tool to monitor Nginx in real time, showing frequent requests, high‑traffic IPs, and providing useful command examples for filtering logs, ordering results, and installing the utility via pip.
On February 10, 2016 a provincial accounting database experienced severe daily‑batch delays despite other applications running normally; a senior DBA collected alert logs, AWR snapshots, ASH dumps, and OSW metrics, uncovered log‑file‑sync and redo‑IO degradation, increased rollbacks, and a faulty SAN link, pinpointing the root cause.
This article shares the author's practical experience building a real‑time log analysis platform at Sina, covering service scope, ELK architecture, performance optimizations, usability improvements, new features, common pitfalls, and a concise Q&A for engineers handling massive log streams.
This article explores how logs serve as vital diagnostic tools in operations, detailing their definition, practical uses for performance and security monitoring, step‑by‑step analysis techniques, and recommendations for free and commercial log analysis solutions.
Learn how to parse web server access logs with AWK, aggregate response times per URL, calculate average latency for each request, and output the results, enabling you to pinpoint long‑running requests that degrade system performance.
Learn three powerful shell commands that reveal the most active IPs, the most requested URLs, and the largest files in an Apache access_log, helping you quickly detect anomalies, optimize resources, and improve site performance.
The article outlines five subtle indicators of a web application breach—abnormal behavior, irregular logs, unexpected processes or users, file modifications, and warning messages—while offering practical monitoring and remediation steps to help security teams detect and mitigate attacks early.
Web Securityapplication monitoringincident response
The article explains why user‑behavior data is often ignored by UED teams, outlines a three‑stage collection‑analysis‑reporting pipeline, and shows how JavaScript running on V8/Node.js can empower front‑end developers to process massive logs, reducing reliance on separate BI resources.
Learn essential log analysis techniques using command‑line tools like grep, cut, and awk, explore advanced filtering with rsyslog and Grok, and discover how log management systems can automate parsing and accelerate troubleshooting of Linux syslog data.
Learn how to use the open‑source Mtools suite to inspect MongoDB logs, extract slow queries, generate performance charts, and overlay system metrics, enabling effective regular health checks, troubleshooting, and optimization for database administrators.
A recent study of five major open‑source distributed systems reveals that most failures can be triggered and reproduced with simple, multi‑event tests, highlighting the importance of systematic testing, deterministic error handling, and concise logging for reliable system operation.
When troubleshooting, you often need to isolate log entries from a precise time window; instead of cumbersome regular expressions, this guide shows how to efficiently capture nginx access logs between two timestamps with a simple awk command, illustrated with an example and a screenshot.