This article explains how NetEase Games leveraged AI, big data, and machine learning to create an AIOps platform that automates anomaly detection, log analysis, and fault localization, improving quality assurance, cost management, and operational efficiency across complex gaming infrastructures.
This guide presents a comprehensive collection of AWK, grep, sed, and netstat one‑liners that let you count unique IPs, rank page visits, measure bandwidth, monitor TCP states, and extract detailed traffic patterns from Apache access logs on Linux systems.
This guide explains how to identify signs of a Linux system intrusion by examining key logs, verifying user account files, monitoring login events, analyzing network traffic, and using lsof to recover deleted security logs, providing step‑by‑step command examples for each task.
HoloInsight is an open‑source, cloud‑native observability platform derived from Ant Group's AntMonitor, offering integrated log‑based monitoring, business metric analysis, and AI‑driven AIOps capabilities while providing a lightweight, modular architecture and extensive extensibility for modern software stacks.
This article presents an AI‑enhanced operations framework that combines metric anomaly detection, alarm compression, log anomaly detection, and intelligent analysis using machine learning methods such as DBSCAN clustering, SARIMAX modeling, Apriori association rules, and LSTM‑based log parsing to improve fault detection and reduce operational costs.
Based on a DTCC2022 presentation, this article explains Apache Doris's high‑performance MPP architecture, its cloud‑native extensions in SelectDB, and how they solve large‑scale log storage and analysis with superior write throughput, storage efficiency, and interactive query speed.
This guide explains how to detect traffic attacks on a low‑traffic Linux website, split Nginx logs daily, identify IPs with excessive requests, and automatically block them using iptables rules scheduled via cron, complete with ready‑to‑run Bash scripts and common firewall commands.
This guide outlines eleven practical steps for Linux system administrators to identify signs of compromise—such as missing logs, altered password files, unusual login activity, abnormal traffic, and deleted files—and provides command examples for detection and recovery.
This article walks through how an online education operator uses Tencent Cloud Log Service to monitor Cloud Connect Network flow logs, set reject‑traffic alerts, analyze regional, IP, and protocol distributions, track bandwidth trends, and finally expand bandwidth to eliminate service interruptions.
A comprehensive collection of practical Linux one‑liners—using awk, grep, sort, uniq, and netstat—to count unique IPs, track page visits, rank URLs, monitor connection states, and measure traffic from Apache access logs for effective server operations.
This article explains why Nginx logs are critical, compares various log‑analysis tools, provides detailed installation and configuration steps for GoAccess, discusses selection criteria, and shares real‑world case studies that demonstrate how to extract valuable system and business insights from massive access logs.
This article walks through the step‑by‑step transformation of a simple online supermarket from a monolithic web app to a fully‑featured microservice architecture, covering common pitfalls, component choices, monitoring, tracing, logging, service discovery, fault‑tolerance, testing, and deployment strategies.
This guide compiles practical Linux shell commands for extracting IP counts, page visit frequencies, time‑range queries, bandwidth usage, HTTP status distribution, and TCP connection states from Apache or Nginx access logs, helping administrators quickly spot traffic patterns, bottlenecks, and potential attacks.
This guide compiles a comprehensive set of Linux command‑line techniques—using awk, grep, netstat, and other tools—to help you count unique IPs, track page visits, identify heavy‑traffic files, monitor connection states, and extract performance metrics from Apache and other web server logs.
The article reviews log‑based, manual, and automated app speed evaluation methods, highlights their trade‑offs, and introduces LazyPerf—a platform that records real‑device interactions, uses resilient widget addressing and built‑in frame detection to dramatically cut automation scripting and calibration effort while improving scalability of performance testing.
This article describes NetEase's AIOps journey for game operations, explaining the Gartner definition of intelligent operations, the implementation roadmap, detailed anomaly‑detection techniques for business, performance, and log data, and a comprehensive fault‑localization workflow that combines resource, code, and historical analysis.
This article investigates why DBLE repeatedly logs "no handler" messages by analyzing DBLE and MySQL logs, using Arthas to trace the call chain, reproducing the scenario with timeout settings, capturing network traffic, and explaining the MySQL 8.0.24 error‑packet behavior that leads to the observed logs.
This article examines security challenges encountered in JD Logistics' sorting platform, details the investigative process for abnormal API requests, proposes a SHA‑256 based authentication scheme with digital signatures, compares industry‑wide API protection methods, and shares practical insights from the author's experience in financial API gateway design.
This article explains how to build a comprehensive monitoring system using the concise USE (Utilization, Saturation, Errors) method, outlines key system and application metrics, and demonstrates practical implementation with Prometheus, Grafana, full‑link tracing, and ELK for observability and performance troubleshooting.
This guide outlines practical steps for Linux system administrators to identify signs of compromise by examining logs, user accounts, login events, network traffic, and recovering deleted files, providing command examples for each investigative task.
The article describes how Ctrip built and continuously improved a Trace system for its ticket‑front‑end microservices, detailing the challenges of distributed logs, the architecture of the solution, and the functional features such as friendly search, multi‑platform aggregation, page replay, and one‑click mock that together boost debugging efficiency for both developers and non‑technical operators.
This article presents seven ready‑to‑use Bash scripts that demonstrate parallel host querying, process statistics gathering, file renaming, directory management, log analysis, network request monitoring, and file size sorting, each with complete code and step‑by‑step explanations.
This article presents a collection of practical Linux shell commands—primarily using awk, grep, sort, uniq, netstat, and tcpdump—to count unique IPs, identify most‑visited pages, filter logs by time or URL, monitor connection states, and calculate traffic statistics for Apache or other web servers.
This guide compiles a comprehensive set of Linux shell commands for analyzing Apache access logs, helping you count unique IPs, identify popular pages, detect suspicious traffic, monitor connection states, measure bandwidth, and extract detailed usage statistics for effective server operations.
This article details the challenges, design choices, deployment steps, detection model creation, data processing, visualization, and future plans of JD Daojia's security operations platform, highlighting the use of Graylog, Elasticsearch, and MongoDB to achieve scalable, real‑time threat detection and response.
This article explains a two‑stage streaming log classification system that uses a prefix‑tree and longest‑common‑subsequence matching to generate log templates, handles traceback logs and cold‑start issues, and combines statistical unsupervised methods to detect anomalies while reducing false alarms.
Learn how to install Kibana, connect it to Elasticsearch, configure index patterns, explore data with Discover, craft queries using Lucene syntax, create visualizations and dashboards, and monitor your stack, all with step‑by‑step guidance and practical examples.
This document describes the background, problems, requirements, analysis, and detailed design of a high‑performance, highly available, and scalable log‑analysis pipeline for MySQL that leverages Kafka, ClickHouse, MySQL, and custom services to aggregate, enrich, and visualize massive query logs.
This article explains how integrating Alibaba Cloud's Log Service (SLS) with the Codeup code‑hosting platform creates a seamless log‑to‑source link, enabling developers and operators to quickly pinpoint problematic code in microservice environments without manual repository setup or local downloads.
This guide presents a comprehensive collection of Linux command‑line techniques—including awk, grep, and netstat—to help you analyze web server logs, identify unique visitors, track page popularity, monitor connection states, and detect performance bottlenecks in a systematic way.
This guide walks through setting up an ELK stack to collect, parse, and visualize Nginx access logs, covering Logstash configuration, Grok patterns, Elasticsearch setup, Nginx proxy with basic authentication, and creating Kibana dashboards for log analysis.
This guide outlines ten practical methods for Linux system administrators to identify signs of intrusion, from missing logs and altered password files to unusual network traffic and recovering deleted log files using process file descriptors.
This step‑by‑step guide shows how to collect, parse, and visualize Nginx access logs using the ELK stack, configure Logstash pipelines, set up Elasticsearch indices, proxy Kibana through Nginx, and secure access with HTTP basic authentication.
This comprehensive guide walks security engineers through every phase of an incident response—from initial information gathering, containment, and vulnerability scanning to detailed log, process, and account analysis, culminating in recovery steps and post‑incident hardening recommendations.
This guide compiles a comprehensive set of Linux shell commands that let you examine web server logs, count unique IPs, identify top‑requested pages, filter bots, monitor connection states, and spot performance or security anomalies, helping you keep your site secure and performant.
Baidu’s Fengjing monitoring platform tackles the daunting challenge of pinpointing failures in its massive Java‑based microservice ecosystem by employing a non‑intrusive probe that captures log metadata, stores it in a database, and reconstructs full request‑level logs with minimal storage overhead.
The Shenzhen Elasticsearch technical event, co‑hosted by the Elastic Chinese community and Tencent Cloud, presented practical sessions on optimizing the Elastic Stack for search, real‑time analytics, logging, security and APM, featuring compression encoding, MongoDB fusion, ByteDance extensions, cost‑effective log storage, Lucene indexing, cross‑cluster replication, vector engine integration, and large‑scale case studies from Tencent, Tiptop Data and vivo.
This article examines frequent HBase RegionServer failures caused by long GC pauses, oversized scans, and HDFS decommissioning, outlines step‑by‑step troubleshooting procedures—including log searches, GC tuning, scan size limits, and monitoring strategies—and provides practical solutions to prevent and resolve these issues.
This article explores the challenges of log aggregation in micro‑service architectures, introduces a globally unique log identifier (logid) with its required properties, compares various logid generation schemes, and presents end‑to‑end solutions for log distribution, aggregation, and analysis using custom tools such as ylog and watcher.
The article introduces the command‑line utility q, explains how to install it on Linux and Windows, and demonstrates using familiar SQLite‑style SQL queries to search, filter, join, and aggregate log files and command output, offering a convenient alternative to traditional shell tools.
This article introduces the command‑line utility q, explains how to install it on Linux and Windows, and demonstrates using full SQLite‑compatible SQL syntax to query, filter, join, and aggregate plain‑text log files, offering a database‑style alternative to traditional shell tools.
This article explains why large‑scale Elasticsearch clusters may miss log entries during keyword searches, dives into the fundamentals of inverted indexes and tokenization, and demonstrates practical index‑time and query‑time tokenizer optimizations—including custom analyzers for English and Chinese—to dramatically improve search recall and precision.
This article walks you through installing and configuring the open‑source WAF‑FLE console—written in PHP—to collect, search, and visualize ModSecurity logs on Apache or Nginx, covering environment setup, database creation, sensor configuration, and troubleshooting common integration issues.
This article explains how to extend a performance testing framework to record and visualize individual HTTP interface latencies by parsing consolidated logs, grouping timings by URL, and using a Groovy script with StatisticsUtil to generate response‑time charts.
This article explains how the HBase hbck tool verifies region consistency and table integrity, lists the most frequently used hbck commands, and describes both low‑risk and high‑risk repair procedures, including handling of RIT states and log‑based troubleshooting.
This guide compiles twenty practical Linux command-line techniques for analyzing web server logs, revealing visitor IP counts, page popularity, request timing, bandwidth usage, error rates, and connection states, enabling administrators to monitor traffic, detect anomalies, and optimize performance.
This guide presents a comprehensive set of Bash and AWK one‑liners for analyzing web server logs, counting unique IPs, tracking page visits, sorting traffic by time, identifying heavy‑weight requests, monitoring Apache processes, and examining network connection states, helping you detect anomalies and optimize performance.
This guide compiles a series of practical Linux one‑liners—using awk, grep, sort, netstat, and related tools—to count unique IPs, rank page requests, filter bots, monitor connection states, and measure bandwidth, enabling quick forensic analysis of Apache or Nginx access logs.
With the shift from monolithic to distributed architectures, this article compares log‑based and network‑data‑based monitoring across data sources, precision, monitoring paths, and implementation methods, concluding that network‑data monitoring offers superior real‑time insight, lower cost, and faster deployment for full‑link observability.
Xianyu’s new Message Quality Platform links client, API, and server logs by a unique messageId, cleans and clusters real‑time telemetry, correlates user behavior, and visualizes abnormal nodes, giving end‑to‑end traceability that cuts incident investigation time by over 90 % and can be applied to other pipelines.
This article demonstrates how red‑team methods can be applied to phishing traceability, detailing phishing classifications, email‑header extraction, malicious site analysis, web‑shell decryption, privilege‑escalation techniques, log mining, and attacker attribution to reconstruct the full attack chain.
This article outlines a request log analysis system that records core request fields, adds proxy‑related data, derives IP‑based ASN and geographic information, parses user‑agent details, and provides comprehensive metrics such as PV/QPS, UV, traffic, latency, status monitoring, and business‑specific insights, all visualized via an ELK‑Kafka architecture.
This article explains practical methods for detecting account security threats—such as blacklisted, expired, or abnormal login behaviors—by analyzing Linux and Windows login logs, defining detection rules, and leveraging automated tools to generate timely alerts and reduce security risks.
This article reviews how major Chinese companies such as JD.com, Ctrip, Quark, 58.com, and Didi have adopted Elasticsearch for large‑scale search, log analysis, and real‑time analytics, detailing their cluster architectures, scaling strategies, and operational practices.
This article explains what MySQL binlog is, how to enable row‑based logging, and provides step‑by‑step Linux commands using mysqlbinlog and grep to extract historical SQL statements such as DELETE operations for troubleshooting and debugging purposes.
This guide explains how to retrieve Hadoop/YARN application logs using the History Server UI, Yarn command‑line tools, and direct HDFS log access, including commands for listing applications, fetching specific logs, and locating the remote log directory.
This article outlines common intrusion signs on CentOS 6.9 systems, explains how to examine logs, user files, login records, network traffic, and demonstrates using lsof and /proc to recover deleted security logs, providing practical commands for Linux security engineers.
This guide explains why Linux is essential for software testers, outlines how to build test environments, use key commands like tail, grep, top, and vmstat for log analysis and performance monitoring, and offers tips for deeper Linux learning and interview preparation.
This tutorial walks backend engineers through real‑world scenarios of log collection, parsing, and analysis using the ELK stack—Logstash, Elasticsearch, and Kibana—showing configuration examples, Grok patterns, RESTful API queries, aggregations, and visualizations to boost operational efficiency.
This article explains why front‑end error logging is essential, how BadJS (JavaScript runtime errors) are captured, enriched, and reported, and provides practical strategies for analyzing logs, handling Script errors, dealing with hybrid WebView environments, and scaling the system with big‑data pipelines.
This comprehensive guide explains why monitoring is essential for operations, outlines monitoring goals and methods, reviews a wide range of open‑source tools, details a Zabbix‑based workflow, enumerates key metrics across hardware, system, application, network, security and business layers, and offers practical alerting and interview tips.
The article explains why loading all log data into a database is impractical, outlines three drawbacks—volatile requests, data bloat, and cost—and introduces the lightweight awk tool with concrete command examples to filter and analyze network logs efficiently without a database.
This guide shows how to generate hourly slow‑query reports from MySQL slow‑log files by using pt‑query‑digest’s timeline feature, filtering with sed, importing the data into SQLite with termsql, and then querying the results to identify hotspot periods and recurring patterns.
This tutorial walks you through installing the GD library, building SARG from source, configuring its key parameters, running reports, using command‑line options, scheduling automated jobs with cron, and troubleshooting common installation and configuration issues for Squid log analysis.
This article presents a modular architecture for real‑time ETL log collection and analysis, detailing the installation and configuration of Flume, Kafka, Storm, Drools, and Redis, and explains how their integration improves fault tolerance, scalability, and processing speed.
When log files grow massive, traditional tools like vim, cat, grep, and awk become slow and memory‑hungry, but Linux’s split command lets you divide a huge file by line count or size, process the pieces individually, and later recombine them, dramatically improving analysis efficiency.
In an interview, Tencent Cloud senior engineer Chen Xi explains how Tencent optimizes Elasticsearch for massive log, monitoring, and document search workloads by prioritizing stability through kernel tweaks, boosting performance with scenario‑specific settings, cutting costs via redundant storage trimming and off‑heap indexing, and leveraging rich data‑pipeline components and robust distributed cluster management to lower operational overhead while anticipating future ecosystem growth and community collaboration.
This guide explains why web log analysis is essential for security, demonstrates how to parse Apache logs, distinguishes normal from malicious requests, and provides practical Secsoso commands for business behavior statistics, traffic monitoring, and detecting attacks such as CC, SQL injection, file inclusion, and XSS.
Monitoring network activity and ensuring compliance requires effective log analysis, and this article reviews five open‑source tools—Graylog, Nagios, Elastic Stack, LOGalyze, and Fluentd—detailing their features, strengths, and use cases for operations and security teams.
This article details Ele.me Logistics' mobile‑app monitoring architecture—E‑Monitor, TimeBomb, Dogger, and EDW—explaining how each layer collects, visualizes, and analyzes business‑level availability data, and showcases a real‑world debugging case that leveraged the stack to resolve an HTTP/2 connectivity bug.
The April 27 Elastic Nanjing Technical Exchange, co‑hosted by Suning Technology and the Elastic Chinese community, showcased how Suning leverages Elasticsearch, Logstash, Kibana and Beats for real‑time search, log analysis, security, and smart‑retail applications, featuring expert case studies and future big‑data strategies.
This article explains how to write two Bash scripts that parse web server access logs with awk and date to automatically list IP addresses that caused failed requests and to list IPs that accessed the site within the last ten minutes.
Pfinder is a new tool that combines problem finder and performance finder functionalities to help users visualize call relationships and time consumption distributions for efficient problem localization in distributed systems.
This guide outlines ten practical methods for Linux administrators to identify compromised machines on CentOS, including log inspection, checking /etc/passwd and /etc/shadow, analyzing login records, monitoring network traffic, and using lsof to recover deleted log files, with step‑by‑step commands and examples.
This article explains the fundamentals of Kubernetes audit logs, their JSON format, recording stages and levels, and shows how to configure policies, analyze logs, and use Alibaba Cloud's integrated solution to create visual reports, custom alerts, and advanced queries for security monitoring.
This guide compiles a comprehensive set of Linux command‑line techniques for parsing Apache and other web server logs, enabling you to count unique IPs, identify hot pages, filter bots, measure bandwidth, track connection states, and spot performance bottlenecks in a single, actionable reference.
This article explains what Android Monkey testing is, walks through the preparation steps, command usage, log redirection, and basic Python-based log filtering, and suggests how to automate the process to improve mobile app stability before release.
This guide compiles a comprehensive set of Linux commands—using awk, grep, netstat, and more—to help you analyze web server logs, track traffic, identify top IPs, monitor connection states, and detect performance bottlenecks on an Alibaba Cloud ECS instance.
This guide walks Linux administrators through common signs of server compromise, shows how to examine logs, user files, active processes, network traffic, and demonstrates using lsof and /proc to recover deleted log files, all with concrete command examples.
The article explains why a downtime diagnosis system is needed, outlines its architecture and implementation methods—including log sources, feature extraction, and API integration—and presents early results showing high automation coverage and significant operational cost savings.
The article explains what a big data development engineer does, the tools and skills required such as Hadoop, Hive, Spark and Kafka, how they process massive logs to compute metrics like PV and UV, and compares this role with conventional business system development.
This guide walks Linux operations engineers through common signs of a breached machine, showing how to examine logs, verify user files, monitor login events, use lsof for deleted files, and recover critical logs with step‑by‑step commands.
This guide explains how to set up a Python‑based log‑analysis tool for web servers, defines key terminology, outlines installation requirements, describes its abstract‑based aggregation features, and demonstrates usage of the accompanying command‑line utility for request, IP, distribution, and detail analysis.
This article details a real-world incident response to a compromised website that redirected visitors to a gambling site, covering intrusion analysis, server remediation, log tracing, and comprehensive cleanup steps to restore security.
This article explains how Elasticsearch and the ELK stack address challenges of storing, securing, retrieving, and analyzing massive data volumes by providing distributed real‑time search, log collection, visualization, and even serving as a NoSQL alternative for large‑scale applications.
Milano, a distributed log collection and analysis platform built on the ELK stack, leverages Filebeat, Kafka, Logstash, Elasticsearch, and Kibana to provide high‑throughput, low‑latency, secure, and visual log management for massive clusters, addressing the challenges of traditional manual log inspection.
This article introduces a Python‑based log analysis tool for web servers that provides minute‑level aggregation, abstracted URI and argument patterns, and multi‑dimensional performance metrics, along with installation steps, core features, implementation details, usage commands, and deployment guidelines.
This article reviews the PreFix system, which uses machine‑learning on datacenter switch logs to predict hardware failures ahead of time, detailing its design, feature extraction, random‑forest model, experimental validation across multiple switch models, and its broader applicability to disk failure prediction.
This guide walks through eleven hands‑on techniques for uncovering UNIX or Solaris intrusions, from inspecting password files and processes to verifying daemon configurations, network sockets, logs, core dumps, hidden files, file integrity, kernel modules, and the limits of manual detection versus IDS solutions.
This article explains how to create a lightweight yet reliable Python‑based log analysis tool that parses nginx logs with regular expressions, stores detailed metrics in MySQL, and provides fine‑grained performance and anomaly reports for web services.
This article explores applying big‑data thinking and platforms—such as Flume, Spark Streaming, and HBase—to operations monitoring, detailing data sources, metric categories, architecture design, implementation steps, and the benefits of a scalable, low‑code monitoring platform.
To mitigate startup latency spikes in Java-based query services caused by class loading, JIT warm‑up, and lazy resource loading, the article presents a generic, low‑cost pre‑warming component that parses local Dubbo and HTTP logs, filters, samples, and replays traffic, detailing its design, implementation, and performance optimizations.
This article outlines the motivation, design principles, architecture, component choices, and step‑by‑step implementation of a comprehensive server security audit system, covering server information collection, log gathering, access control checks, local vulnerability detection, abnormal traffic analysis, and integration with ELK, Hadoop, and open‑source tools like Lynis and OSSEC.
This article presents five practical Python scripts that demonstrate how to traverse directories and list files, draw text triangles, create a number‑guessing game, log disk usage, and analyze IP access counts from log files, each explained with clear code examples.
This article explains how to analyze and visualize Docker container logs using the ELK stack, covering preparation, parsing tips, Kibana query techniques, and example visualizations to help monitor Dockerized environments effectively in production.
This article explains Google’s BeyondCorp concept, the need for deep defense of internal and perimeter networks, and provides practical Linux scripts for monitoring processes, ports, command usage, system events, file changes, and SFTP activity to detect and mitigate host intrusions.
Alibaba’s SunFire platform delivers massive‑scale, real‑time log collection, processing, and visualization for e‑commerce spikes like Double 11, using low‑overhead agents, asynchronous Map/Reduce pipelines, fault‑tolerant task scheduling, and shared inputs to ensure accurate, low‑latency monitoring across billions of transactions.
The article critiques popular but unnecessary log‑analysis features—such as sub‑second alerts, endless pagination, flashy maps, full SQL support, bulk downloads, and live tail—arguing that focusing on practical alert content, efficient querying, and proper architecture yields far more value for IT operations.
This article describes Ctrip's challenges with web security, evaluates hardware and commercial cloud WAF shortcomings, and presents a low‑cost, low‑risk cloud‑based WAF solution that leverages DNS redirection, closed‑loop rule management, Lua/Tengine deployment, supervised machine‑learning log analysis, and big‑data streaming for real‑time threat detection and mitigation.
This tutorial shows how to quickly set up a log analysis platform using the ELK stack, OpenResty, Ansible, and Vagrant, covering architecture, required tools, server provisioning, deployment commands, testing procedures, and next‑step enhancements for real‑world use.
