Tagged articles

49 articles

Page 1 of 1

Sep 17, 2025 · Backend Development

4 Essential Message Queue Use Cases Every Backend Engineer Should Master

Message queues are foundational to large‑scale architectures, and this article explains four key scenarios—asynchronous communication, application decoupling, flash‑sale traffic buffering, and log processing—illustrated with diagrams to help backend engineers design resilient, high‑throughput systems.

Log ProcessingMessage QueueSystem Decoupling

0 likes · 4 min read

4 Essential Message Queue Use Cases Every Backend Engineer Should Master

Alibaba Cloud Observability

Sep 15, 2025 · Information Security

How Alibaba Cloud’s New mask Function Boosts Log Data Security and Performance

This article explains why data desensitization is now a compliance must‑have, reviews Alibaba Cloud Log Service’s existing masking pipelines, introduces the new mask function with its keyword and built‑in modes, compares its performance against regex solutions, and showcases three real‑world use cases covering transaction logs, large‑model interactions, and Nginx URI parameters.

AILog ProcessingSLS

0 likes · 12 min read

How Alibaba Cloud’s New mask Function Boosts Log Data Security and Performance

Alibaba Cloud Native

Sep 10, 2025 · Information Security

How Alibaba Cloud SLS’s New mask Function Simplifies Large‑Scale Log Desensitization

In the AI era, massive interaction data drives rapid smart‑app growth, but personal privacy risks demand robust data‑masking; Alibaba Cloud Log Service (SLS) introduces a versatile mask function that replaces complex regex pipelines with concise configurations, boosting performance, reducing maintenance, and meeting strict compliance such as GDPR and China’s Personal Information Protection Law.

Cloud NativeLog ProcessingSLS

0 likes · 12 min read

How Alibaba Cloud SLS’s New mask Function Simplifies Large‑Scale Log Desensitization

php Courses

Aug 29, 2025 · Operations

How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights

Learn how to transform PHP logs into real‑time, structured events by implementing a log event pipeline that includes JSON logging, lightweight collectors like Filebeat, streaming platforms such as Kafka or Flink, enrichment, and visualization with Grafana, enabling instant monitoring, alerting, and data‑driven decisions.

FlinkGrafanaKafka

0 likes · 7 min read

How to Build a Real‑Time PHP Log Event Pipeline for Instant Insights

Alibaba Cloud Observability

Jul 14, 2025 · Cloud Native

How to Tame Massive New‑Energy‑Vehicle Logs with Cloud‑Native SLS Data Cleaning

This article explains the five major challenges of handling heterogeneous, weakly‑structured log data in the new‑energy‑vehicle ecosystem and demonstrates how Alibaba Cloud's Log Service (SLS) provides cloud‑native, real‑time data cleaning, cross‑region aggregation, cost‑optimized storage, and visual lineage to enable comprehensive operational insight and intelligent management.

Log ProcessingSLSdata cleaning

0 likes · 18 min read

How to Tame Massive New‑Energy‑Vehicle Logs with Cloud‑Native SLS Data Cleaning

Alibaba Cloud Observability

Jul 14, 2025 · Operations

How New SPL Operators Supercharge Log Processing Performance

The latest SPL update introduces powerful operators like pack-fields, log-to-metric, and metric-to-metric, delivering dramatic performance gains, richer data transformation capabilities, and enhanced observability for cloud‑native log processing pipelines.

Log ProcessingSPLdata pipeline

0 likes · 8 min read

How New SPL Operators Supercharge Log Processing Performance

Alibaba Cloud Native

Jul 9, 2025 · Cloud Native

How SPL’s New Operators Supercharge Log‑to‑Metric Processing in Cloud‑Native Environments

The article introduces SPL’s latest operators—pack-fields, log-to-metric, and metric-to-metric—explaining their smart field aggregation, trimming, type inference, wildcard matching, and label manipulation capabilities, and demonstrates through code examples and performance benchmarks how they dramatically improve data processing efficiency and observability in cloud‑native log services.

Cloud NativeLog ProcessingSPL

0 likes · 10 min read

How SPL’s New Operators Supercharge Log‑to‑Metric Processing in Cloud‑Native Environments

Lobster Programming

Jan 16, 2025 · Big Data

How to Extract Top 100 Search Keywords from Billion‑Scale Logs Efficiently

This article explains a divide‑and‑conquer method that splits massive search‑log files, uses multithreaded hashing to count keyword frequencies, and applies a min‑heap to efficiently retrieve the top‑100 most frequent search terms for SEO and recommendation tasks.

Big DataHashingLog Processing

0 likes · 3 min read

How to Extract Top 100 Search Keywords from Billion‑Scale Logs Efficiently

Alibaba Cloud Observability

Nov 13, 2024 · Cloud Native

Master Log Processing with iLogtail SPL: From Native Plugins to Advanced Transformations

This article introduces SLS's SPL (SLS Processing Language) for iLogtail, compares its three processing modes, details feature differences, highlights the advantages of iLogtail 2.0 + SPL, and provides step‑by‑step SPL examples covering regex, delimiter, JSON parsing, desensitization, field addition, encoding, mathematical operations, URL handling, and logical comparisons.

Log ProcessingSPLdata transformation

0 likes · 20 min read

Master Log Processing with iLogtail SPL: From Native Plugins to Advanced Transformations

Alibaba Cloud Native

Oct 16, 2024 · Cloud Native

Master Log Processing with iLogtail SPL: From Native Plugins to Advanced Transformations

This guide explains how iLogtail 2.0 introduces the SPL (SLS Processing Language) to unify log and time‑series data handling, compares native, extension, and SPL processing modes, and provides step‑by‑step SPL examples for regex, delimiter, JSON, desensitization, field addition, encoding, URL parsing, and mathematical operations.

Cloud NativeLog ProcessingSPL

0 likes · 17 min read

Alibaba Cloud Observability

Aug 12, 2024 · Cloud Native

Mastering Log Data Cleaning with SPL: From Error Filtering to Time Parsing

This article compares the new SPL syntax with the legacy DSL for SLS log processing, demonstrating how to filter errors, perform fuzzy matches, handle numeric ranges, manage fields, apply conditional expressions, parse timestamps, and extract data from unstructured sources using concise examples.

Log ProcessingSLSSPL

0 likes · 10 min read

Mastering Log Data Cleaning with SPL: From Error Filtering to Time Parsing

Alibaba Cloud Native

Aug 5, 2024 · Operations

Mastering SPL vs DSL: Practical Data Cleaning and Transformation Techniques

This guide compares the new SPL syntax with the legacy DSL for common data processing tasks such as error log filtering, field management, time parsing, and extracting unstructured data, providing concrete command examples for each scenario.

DSLLog ProcessingSPL

0 likes · 10 min read

Mastering SPL vs DSL: Practical Data Cleaning and Transformation Techniques

Alibaba Cloud Observability

Jul 31, 2024 · Cloud Native

How the New SLS Data Processing Boosts Performance, Cuts Cost, and Simplifies Debugging with SPL

This article explains how Alibaba Cloud's SLS data processing resolves the tension between simple log collection and the need for structured, analyzable data by introducing a unified SPL syntax, delivering over tenfold performance gains, reducing costs to one‑third, and providing powerful debugging tools for cloud‑native log analytics.

Log ProcessingSPLdata engineering

0 likes · 8 min read

How the New SLS Data Processing Boosts Performance, Cuts Cost, and Simplifies Debugging with SPL

Alibaba Cloud Observability

Jun 27, 2024 · Operations

iLogtail SPL vs Logstash: Faster, Lighter, More Flexible Log Processing

iLogtail 2.0 introduces an SPL processing mode that outperforms Logstash’s filter plugins in functionality, resource consumption, and throughput across multiple test scenarios, offering lower CPU and memory usage, faster start‑up, and superior handling of complex JSON and high‑volume log streams.

Log ProcessingSPLiLogtail

0 likes · 17 min read

iLogtail SPL vs Logstash: Faster, Lighter, More Flexible Log Processing

Alibaba Cloud Observability

Apr 16, 2024 · Cloud Native

Mastering Interactive Log Exploration with SPL: Unix‑Inspired Pipelines in Cloud Native Environments

This article explains how the SLS Processing Language (SPL) brings Unix‑style pipelined, interactive log exploration to cloud‑native observability, detailing why logs are unstructured, how SPL’s unified syntax works, and which commands simplify field projection, enrichment, filtering, and semi‑structured data parsing.

Log ProcessingSPLUnix pipes

0 likes · 12 min read

Mastering Interactive Log Exploration with SPL: Unix‑Inspired Pipelines in Cloud Native Environments

Volcano Engine Developer Services

Nov 16, 2023 · Big Data

Why Replace Logstash with Flink? Boost Log Processing Performance and Reliability

This article examines the shortcomings of Logstash in log collection—data loss, poor performance, high troubleshooting cost, and lack of dynamic scaling—and demonstrates how migrating to Flink can provide at‑least‑once semantics, flexible error handling, high‑throughput low‑latency processing, automatic resource scaling, and advanced analytics within the ELK ecosystem.

Data StreamingELKFlink

0 likes · 9 min read

Why Replace Logstash with Flink? Boost Log Processing Performance and Reliability

Tencent Advertising Technology

Dec 27, 2022 · Big Data

Design and Optimization of Tencent Advertising Log Data Lake Using Iceberg, Spark, and Flink

The article details how Tencent Advertising re‑architected its massive log pipeline by consolidating heterogeneous real‑time and offline logs into an Iceberg‑based data lake, introducing multi‑level partitioning, Spark and Flink ingestion, and numerous performance and cost optimizations for scalable big‑data analytics.

Big DataData LakeFlink

0 likes · 20 min read

Design and Optimization of Tencent Advertising Log Data Lake Using Iceberg, Spark, and Flink

21CTO

Nov 20, 2022 · Big Data

How Meituan’s Logan Real‑Time Log System Boosts Debugging Across Mobile, Web, and IoT

This article details the design, architecture, and implementation of Meituan's Logan real‑time logging platform, covering its workflow, multi‑terminal collection SDK, ingestion, Flink‑based processing, consumption layers, stability measures, and future roadmap, illustrating how it improves fault diagnosis and system reliability.

ElasticsearchFlinkKafka

0 likes · 18 min read

How Meituan’s Logan Real‑Time Log System Boosts Debugging Across Mobile, Web, and IoT

DataFunSummit

May 21, 2022 · Big Data

Tencent News Massive Log Processing Architecture and Data Applications

The article presents Tencent News' comprehensive massive log processing solution, covering background, overall architecture, data collection, real-time and offline computation layers, data quality assurance, and practical examples such as Flink CDC for database synchronization, illustrating how large‑scale data is managed and applied.

FlinkLog ProcessingTencent

0 likes · 10 min read

Tencent News Massive Log Processing Architecture and Data Applications

Qingyun Technology Community

Sep 2, 2021 · Cloud Native

How to Build a Serverless Log Processing Pipeline with OpenFunction and Kafka

This guide explains how to collect container logs with Kafka, deploy OpenFunction to process them serverlessly, write a Go handler that detects 404 errors, and send alerts to Slack, demonstrating a cost‑effective, scalable log‑processing solution on Kubernetes.

KafkaKubernetesLog Processing

0 likes · 16 min read

How to Build a Serverless Log Processing Pipeline with OpenFunction and Kafka

21CTO

Jun 16, 2021 · Backend Development

How to Process a 16 GB Log File in Seconds with Go Concurrency

This article explains how to efficiently extract time‑range logs from a massive 16 GB .txt/.log file using Go's bufio.NewReader, sync.Pool for buffer reuse, and concurrent goroutines, achieving processing times of around 25 seconds.

Log Processingconcurrencylarge files

0 likes · 9 min read

How to Process a 16 GB Log File in Seconds with Go Concurrency

MaGe Linux Operations

Jun 14, 2021 · Backend Development

How to Process a 16 GB Log File in Seconds with Go

Learn how to efficiently extract timestamped logs from a massive 16 GB file in seconds using Go's buffered I/O, sync.Pool, and goroutine concurrency, with step‑by‑step code examples, performance tips, and a complete runnable program.

Log Processingconcurrencylarge files

0 likes · 10 min read

How to Process a 16 GB Log File in Seconds with Go

Architecture Digest

Jun 10, 2021 · Big Data

NetEase Game Streaming ETL Architecture and Practices Based on Flink

This article presents NetEase Game's streaming ETL solution built on Flink, covering business background, log characteristics, specialized and generic ETL services, architectural evolution, Python UDF integration, runtime optimizations, fault‑tolerance mechanisms, and future roadmap for unified real‑time and offline data warehouses.

Big DataFlinkLog Processing

0 likes · 19 min read

NetEase Game Streaming ETL Architecture and Practices Based on Flink

IT Architects Alliance

Apr 20, 2021 · Big Data

Real-time Log Processing System Based on Flink and Drools

This article describes a real-time log processing platform that integrates Kafka, Flink, Drools rule engine, Redis, and Elasticsearch to unify heterogeneous log formats, extract business metrics, and provide configurable, dynamic data processing for large‑scale logging scenarios.

DroolsElasticsearchFlink

0 likes · 6 min read

Real-time Log Processing System Based on Flink and Drools

Liangxu Linux

Feb 25, 2021 · Fundamentals

Quick Shell Tricks to Remove Blank Lines from Logs with grep, sed, awk, tr, and vi

This article shares concise, easy-to-understand shell command techniques—using grep, sed, awk, tr, and vi—to efficiently delete empty lines from log files, highlighting each tool's syntax, options, and practical usage for embedded software developers.

GrepLog ProcessingShell

0 likes · 5 min read

Quick Shell Tricks to Remove Blank Lines from Logs with grep, sed, awk, tr, and vi

dbaplus Community

Jan 5, 2021 · Big Data

How Ctrip Built a Scalable Unified Log Framework for Payment Data

Facing massive, heterogeneous logs from numerous payment services, Ctrip’s data team designed a unified logging framework that extends log4j2, streams logs via Kafka to HDFS using a customized Camus pipeline, partitions and stores data in ORC for efficient Hive analysis, while addressing format, storage, and performance challenges.

Big DataCamusHadoop

0 likes · 16 min read

How Ctrip Built a Scalable Unified Log Framework for Payment Data

MaGe Linux Operations

Sep 15, 2020 · Fundamentals

Why Python Can Replace Complex Shell Scripts: A Practical Linux Log Example

This article explains how Linux command‑line pipelines work, highlights the limitations of traditional shell scripts, and demonstrates how a concise Python script can replace them for counting unique users in log files while remaining easy to read, reusable, and powerful.

LinuxLog ProcessingPython

0 likes · 9 min read

Why Python Can Replace Complex Shell Scripts: A Practical Linux Log Example

Ctrip Technology

Sep 10, 2020 · Big Data

Design and Implementation of a Unified Log Framework for Ctrip Payment Center

The article describes the design, architecture, and operational details of a unified logging framework at Ctrip's payment center, covering log production via a Log4j2 extension, Kafka‑Camus collection, Hive/ORC storage, MapReduce parsing optimizations, and governance strategies for massive daily TB‑scale data.

Big DataCamusData Governance

0 likes · 15 min read

Design and Implementation of a Unified Log Framework for Ctrip Payment Center

Programmer DD

Jul 23, 2020 · Backend Development

Why Message Queues Are Essential for Scalable Backend Architecture

This article explains the role of message‑queue middleware in distributed systems, covering its core functions, common use cases such as asynchronous processing, application decoupling, traffic shaping, log handling and messaging, and provides concrete architectural examples illustrating performance improvements and design patterns.

Backend ArchitectureLog ProcessingTraffic Shaping

0 likes · 9 min read

Why Message Queues Are Essential for Scalable Backend Architecture

Ctrip Technology

Jan 22, 2020 · Databases

Migrating Log Processing from Elasticsearch to ClickHouse: Architecture, Deployment, Optimization, and Benefits

This article details Ctrip's migration of large‑scale log processing from Elasticsearch to ClickHouse, explaining why ClickHouse was chosen, the high‑availability deployment architecture, data ingestion strategies, dashboard integration, performance gains, operational practices, and overall cost and reliability improvements.

Distributed SystemsElasticsearchLog Processing

0 likes · 12 min read

Migrating Log Processing from Elasticsearch to ClickHouse: Architecture, Deployment, Optimization, and Benefits

FunTester

Oct 4, 2019 · Backend Development

Python Utility for Reading Log Files and Generating Violin Charts with Plotly for Server Performance Testing

This article presents a Python script that reads performance test log files, extracts timing data, and visualizes the results as violin charts using Plotly, providing a reusable utility for server performance analysis and reporting.

Data visualizationLog Processingviolin chart

0 likes · 4 min read

Python Utility for Reading Log Files and Generating Violin Charts with Plotly for Server Performance Testing

Architecture Digest

Sep 24, 2019 · Big Data

Implementation Principles and Architecture of DBus Data Sources (RDBMS and Log Types)

The article explains how DBus ingests data from relational databases and log sources by detailing its extractor, incremental conversion, and full‑pull modules, the use of Canal and Kafka, rule‑based log structuring, the unified UMS message format, and heartbeat monitoring for reliability.

CanalDBusKafka

0 likes · 13 min read

Implementation Principles and Architecture of DBus Data Sources (RDBMS and Log Types)

Youzan Coder

Aug 14, 2019 · Big Data

Comprehensive Guide to Data Collection, Event Modeling, and Tracking in Big Data Platforms

The guide explains how comprehensive data collection in big‑data platforms relies on a standardized event model, passive and code‑based embedding, multi‑platform SDKs, a log‑middleware layer, precise location tracking, and an embedding management platform that supports workflow, testing, quality monitoring, and scalable infrastructure for future enhancements.

AnalyticsBig DataLog Processing

0 likes · 19 min read

Comprehensive Guide to Data Collection, Event Modeling, and Tracking in Big Data Platforms

NetEase Game Operations Platform

Aug 4, 2019 · Big Data

Log Classification and Real-Time Aggregation Architecture Using Flink and Kafka

This article describes a real‑time log‑classification pipeline built on Flink and Kafka that pre‑filters, structures, classifies, and aggregates heterogeneous logs, enabling efficient frequency‑based alerts and statistical analysis without storing raw log data at scale.

FlinkKafkaLog Processing

0 likes · 11 min read

Log Classification and Real-Time Aggregation Architecture Using Flink and Kafka

Java Captain

Jun 29, 2018 · Backend Development

Introduction to Message Queue Middleware and Its Application Scenarios

This article introduces message queue middleware, explains its role in distributed systems for asynchronous processing, system decoupling, traffic shaping, log handling and message communication, and provides concrete e‑commerce and log‑collection examples illustrating how queues improve performance, scalability and reliability.

Log ProcessingMessage QueueSystem Decoupling

0 likes · 8 min read

Introduction to Message Queue Middleware and Its Application Scenarios

Java Backend Technology

May 6, 2018 · Backend Development

How Message Queues Boost Performance and Decouple Applications

This article explains how message queue middleware improves distributed systems by enabling asynchronous processing, decoupling services, handling traffic spikes, aggregating logs, and supporting various communication patterns, illustrated with real‑world e‑commerce and logging architectures that boost throughput and reliability.

Log ProcessingTraffic Shapingapplication decoupling

0 likes · 9 min read

How Message Queues Boost Performance and Decouple Applications

dbaplus Community

Dec 26, 2017 · Big Data

Turning Raw Logs into Structured Data with DBus Visual Rule Operators

This article explains how the open‑source DBus platform, combined with the Wormhole streaming engine, captures raw application logs, lets users configure visual rule operators, and transforms the unstructured message part into schema‑driven, Kafka‑ready data for downstream analytics.

Big DataDBusLog Processing

0 likes · 15 min read

Turning Raw Logs into Structured Data with DBus Visual Rule Operators

Efficient Ops

Dec 5, 2017 · Operations

How Alibaba’s Sunfire Achieves Second‑Level Monitoring at Trillion‑Transaction Scale

This article explains how Alibaba’s Sunfire monitoring platform processes terabytes of logs per minute, uses a pull‑based architecture with Brain‑Reduce‑Map roles, tackles scalability and reliability challenges, and outlines future directions such as MQL standardization and intelligent baselines.

Log ProcessingOperationsReal-Time

0 likes · 17 min read

How Alibaba’s Sunfire Achieves Second‑Level Monitoring at Trillion‑Transaction Scale

21CTO

Aug 5, 2017 · Backend Development

How I Reduced Log Keyword Counting from Hours to Minutes Using PHP, Grep, Regex & Trie

This article walks through solving a massive log‑keyword counting task—600,000 short messages and 50,000 keywords—by evolving from a simple grep‑based approach to regex optimizations, word‑splitting, a trie data structure, and finally a multi‑process Redis queue, achieving a performance boost from hours to under ten minutes.

GrepLog ProcessingTrie

0 likes · 15 min read

How I Reduced Log Keyword Counting from Hours to Minutes Using PHP, Grep, Regex & Trie

Tongcheng Travel Technology Center

Apr 10, 2017 · Operations

Sentinel Monitoring System: Real‑Time Business Log Monitoring and Incident Detection for an Airline Ticket Platform

The Sentinel system was built to provide real‑time, zero‑modification monitoring of airline ticket business services by consuming Tianwang logs through a Storm cluster, offering flexible rule configuration, addressing performance pitfalls, and planning future enhancements such as custom monitoring scripts and visual dashboards.

KafkaLog ProcessingReal-Time

0 likes · 6 min read

Sentinel Monitoring System: Real‑Time Business Log Monitoring and Incident Detection for an Airline Ticket Platform

360 Quality & Efficiency

Mar 17, 2017 · Backend Development

Using jq for JSON Log Extraction and Real‑Time Monitoring on the Command Line

This tutorial introduces the jq command‑line tool, shows how to download and install it, and demonstrates practical commands for extracting specific JSON fields from log data, chaining jq pipelines, and monitoring click and PV metrics in real time.

Log Processingjq

0 likes · 6 min read

Using jq for JSON Log Extraction and Real‑Time Monitoring on the Command Line

ITFLY8 Architecture Home

Feb 24, 2017 · Big Data

How ELK, Kafka, and Spark Streaming Revolutionize Log Management in Big Data Environments

This article explores the evolution of log processing in the big‑data era, detailing how ELK Stack, Kafka, and Spark Streaming work together to provide scalable, real‑time log collection, analysis, and visualization for modern cloud‑native operations.

Big DataELKKafka

0 likes · 12 min read

How ELK, Kafka, and Spark Streaming Revolutionize Log Management in Big Data Environments

dbaplus Community

Feb 6, 2017 · Operations

How JD’s CallGraph Transforms Distributed Tracing for Real‑Time Operations

CallGraph, JD.com’s in‑house distributed tracing platform, provides low‑intrusion, high‑performance monitoring for micro‑service ecosystems, enabling real‑time call‑graph analysis, TP metrics, flexible configuration, and future extensions such as deep‑learning‑driven insights.

Distributed TracingLog Processingmonitoring

0 likes · 15 min read

How JD’s CallGraph Transforms Distributed Tracing for Real‑Time Operations

Architecture Digest

Sep 21, 2016 · Big Data

Log Platform Architecture and Scaling Lessons from Vipshop's 419 Promotion

This article presents a detailed case study of Vipshop's log platform during the 419 sales event, analyzing the 2013 architecture, bottlenecks in RabbitMQ and Storm, and the subsequent redesign using Kafka, Impala, and HBase to achieve scalable, reliable big‑data processing.

Big DataImpalaKafka

0 likes · 16 min read

Log Platform Architecture and Scaling Lessons from Vipshop's 419 Promotion

Architecture Digest

Sep 14, 2016 · Backend Development

Log Platform Architecture and Scaling Lessons from Vipshop’s 419 Flash Sale

The article analyzes Vipshop’s 419 flash‑sale log platform, detailing the 2013 architecture using Flume, RabbitMQ, Storm, Redis and MySQL, diagnosing bottlenecks in RabbitMQ and Storm during traffic spikes, and presenting practical scaling and monitoring solutions for high‑throughput backend systems.

Log ProcessingRabbitMQScalability

0 likes · 8 min read

Log Platform Architecture and Scaling Lessons from Vipshop’s 419 Flash Sale

Java High-Performance Architecture

Feb 25, 2016 · Backend Development

Understanding Kafka: Architecture, Use Cases, and Core Components

This article explains Kafka's high‑throughput distributed messaging architecture, its typical scenarios such as log and operational data collection, the roles of producers, brokers, topics, consumers, Zookeeper, and provides a practical example of detecting abnormal user transactions.

Data StreamingKafkaLog Processing

0 likes · 3 min read

Understanding Kafka: Architecture, Use Cases, and Core Components

Java High-Performance Architecture

Dec 26, 2015 · Backend Development

How to Use Redis Queues for High-Concurrency Log Processing

Learn how to handle high‑concurrency log collection by buffering logs in a Redis queue, then periodically batch‑inserting them into a database, with code examples for enqueuing, processing, error rollback, and scheduling the task.

Batch InsertLog ProcessingMessage Queue

0 likes · 3 min read

How to Use Redis Queues for High-Concurrency Log Processing

21CTO

Nov 4, 2015 · Big Data

How We Built a Real‑Time Log Analytics Platform with Storm and Cardinality Counting

To monitor hundreds of web apps on UAE’s PaaS platform in near‑real time, we combined Storm with lightweight log transport, a memcached‑based fqueue, and adaptive cardinality counting to efficiently compute PV, UV, response times, and custom metrics while handling cross‑cluster log aggregation.

Big DataCardinality countingLog Processing

0 likes · 9 min read

How We Built a Real‑Time Log Analytics Platform with Storm and Cardinality Counting

MaGe Linux Operations

Nov 3, 2015 · Operations

Master AWK for Log Analysis: A Quick Beginner’s Guide

This tutorial walks beginners through essential AWK commands and techniques for parsing and filtering log files, covering field extraction, separators, arithmetic on string fields, BEGIN/END blocks, conditional filters, external parameters, and common functions with practical examples.

Log ProcessingSysadminawk

0 likes · 11 min read

Master AWK for Log Analysis: A Quick Beginner’s Guide