Tagged articles
log processing
50 articles · Page 1 of 1
This article compares JSONL (JSON Lines) with regular JSON, explaining how line-delimited records enable memory-efficient streaming for logs, machine learning datasets, and ETL pipelines, while detailing tool support in pandas, Spark, and Hugging Face, and clarifying when to stick with standard JSON.
Apache SparkETLHugging Face
0 likes · 21 min read

Message queues are foundational to large‑scale architectures, and this article explains four key scenarios—asynchronous communication, application decoupling, flash‑sale traffic buffering, and log processing—illustrated with diagrams to help backend engineers design resilient, high‑throughput systems.
Message QueueSystem Decouplingasynchronous communication
0 likes · 4 min read

This article explains why data desensitization is now a compliance must‑have, reviews Alibaba Cloud Log Service’s existing masking pipelines, introduces the new mask function with its keyword and built‑in modes, compares its performance against regex solutions, and showcases three real‑world use cases covering transaction logs, large‑model interactions, and Nginx URI parameters.
AISLScloud security
0 likes · 12 min read

In the AI era, massive interaction data drives rapid smart‑app growth, but personal privacy risks demand robust data‑masking; Alibaba Cloud Log Service (SLS) introduces a versatile mask function that replaces complex regex pipelines with concise configurations, boosting performance, reducing maintenance, and meeting strict compliance such as GDPR and China’s Personal Information Protection Law.
SLScloud nativedata masking
0 likes · 12 min read

Learn how to transform PHP logs into real‑time, structured events by implementing a log event pipeline that includes JSON logging, lightweight collectors like Filebeat, streaming platforms such as Kafka or Flink, enrichment, and visualization with Grafana, enabling instant monitoring, alerting, and data‑driven decisions.
FlinkGrafanaKafka
0 likes · 7 min read

This article explains the five major challenges of handling heterogeneous, weakly‑structured log data in the new‑energy‑vehicle ecosystem and demonstrates how Alibaba Cloud's Log Service (SLS) provides cloud‑native, real‑time data cleaning, cross‑region aggregation, cost‑optimized storage, and visual lineage to enable comprehensive operational insight and intelligent management.
SLSdata cleaninglog processing
0 likes · 18 min read

The latest SPL update introduces powerful operators like pack-fields, log-to-metric, and metric-to-metric, delivering dramatic performance gains, richer data transformation capabilities, and enhanced observability for cloud‑native log processing pipelines.
SPLdata pipelinelog processing
0 likes · 8 min read

The article introduces SPL’s latest operators—pack-fields, log-to-metric, and metric-to-metric—explaining their smart field aggregation, trimming, type inference, wildcard matching, and label manipulation capabilities, and demonstrates through code examples and performance benchmarks how they dramatically improve data processing efficiency and observability in cloud‑native log services.
SPLcloud nativedata pipelines
0 likes · 10 min read

This article explains a divide‑and‑conquer method that splits massive search‑log files, uses multithreaded hashing to count keyword frequencies, and applies a min‑heap to efficiently retrieve the top‑100 most frequent search terms for SEO and recommendation tasks.
Multithreadingbig datahashing
0 likes · 3 min read

This article introduces SLS's SPL (SLS Processing Language) for iLogtail, compares its three processing modes, details feature differences, highlights the advantages of iLogtail 2.0 + SPL, and provides step‑by‑step SPL examples covering regex, delimiter, JSON parsing, desensitization, field addition, encoding, mathematical operations, URL handling, and logical comparisons.
SPLdata transformationiLogtail
0 likes · 20 min read

This guide explains how iLogtail 2.0 introduces the SPL (SLS Processing Language) to unify log and time‑series data handling, compares native, extension, and SPL processing modes, and provides step‑by‑step SPL examples for regex, delimiter, JSON, desensitization, field addition, encoding, URL parsing, and mathematical operations.
SPLcloud nativedata transformation
0 likes · 17 min read

This article compares the new SPL syntax with the legacy DSL for SLS log processing, demonstrating how to filter errors, perform fuzzy matches, handle numeric ranges, manage fields, apply conditional expressions, parse timestamps, and extract data from unstructured sources using concise examples.
SLSSPLdata cleaning
0 likes · 10 min read

This guide compares the new SPL syntax with the legacy DSL for common data processing tasks such as error log filtering, field management, time parsing, and extracting unstructured data, providing concrete command examples for each scenario.
DSLSPLdata cleaning
0 likes · 10 min read

This article explains how Alibaba Cloud's SLS data processing resolves the tension between simple log collection and the need for structured, analyzable data by introducing a unified SPL syntax, delivering over tenfold performance gains, reducing costs to one‑third, and providing powerful debugging tools for cloud‑native log analytics.
Data EngineeringSPLdebugging
0 likes · 8 min read

iLogtail 2.0 introduces an SPL processing mode that outperforms Logstash’s filter plugins in functionality, resource consumption, and throughput across multiple test scenarios, offering lower CPU and memory usage, faster start‑up, and superior handling of complex JSON and high‑volume log streams.
ObservabilitySPLiLogtail
0 likes · 17 min read

This article explains how the SLS Processing Language (SPL) brings Unix‑style pipelined, interactive log exploration to cloud‑native observability, detailing why logs are unstructured, how SPL’s unified syntax works, and which commands simplify field projection, enrichment, filtering, and semi‑structured data parsing.
ObservabilitySPLUnix pipes
0 likes · 12 min read

This article examines the shortcomings of Logstash in log collection—data loss, poor performance, high troubleshooting cost, and lack of dynamic scaling—and demonstrates how migrating to Flink can provide at‑least‑once semantics, flexible error handling, high‑throughput low‑latency processing, automatic resource scaling, and advanced analytics within the ELK ecosystem.
Data StreamingFlinkLogstash
0 likes · 9 min read

The article details how Tencent Advertising re‑architected its massive log pipeline by consolidating heterogeneous real‑time and offline logs into an Iceberg‑based data lake, introducing multi‑level partitioning, Spark and Flink ingestion, and numerous performance and cost optimizations for scalable big‑data analytics.
FlinkSparkbig data
0 likes · 20 min read


21CTONov 20, 2022 · Big Data
This article details the design, architecture, and implementation of Meituan's Logan real‑time logging platform, covering its workflow, multi‑terminal collection SDK, ingestion, Flink‑based processing, consumption layers, stability measures, and future roadmap, illustrating how it improves fault diagnosis and system reliability.
ElasticsearchFlinkKafka
0 likes · 18 min read

The article presents Tencent News' comprehensive massive log processing solution, covering background, overall architecture, data collection, real-time and offline computation layers, data quality assurance, and practical examples such as Flink CDC for database synchronization, illustrating how large‑scale data is managed and applied.
FlinkTencentdata pipeline
0 likes · 10 min read

This guide explains how to collect container logs with Kafka, deploy OpenFunction to process them serverlessly, write a Go handler that detects 404 errors, and send alerts to Slack, demonstrating a cost‑effective, scalable log‑processing solution on Kubernetes.
KafkaKubernetesOpenFunction
0 likes · 16 min read


21CTOJun 16, 2021 · Backend Development
This article explains how to efficiently extract time‑range logs from a massive 16 GB .txt/.log file using Go's bufio.NewReader, sync.Pool for buffer reuse, and concurrent goroutines, achieving processing times of around 25 seconds.
ConcurrencyLarge Fileslog processing
0 likes · 9 min read

Learn how to efficiently extract timestamped logs from a massive 16 GB file in seconds using Go's buffered I/O, sync.Pool, and goroutine concurrency, with step‑by‑step code examples, performance tips, and a complete runnable program.
ConcurrencyLarge Fileslog processing
0 likes · 10 min read

This article presents NetEase Game's streaming ETL solution built on Flink, covering business background, log characteristics, specialized and generic ETL services, architectural evolution, Python UDF integration, runtime optimizations, fault‑tolerance mechanisms, and future roadmap for unified real‑time and offline data warehouses.
FlinkStreaming ETLbig data
0 likes · 19 min read

This article describes a real-time log processing platform that integrates Kafka, Flink, Drools rule engine, Redis, and Elasticsearch to unify heterogeneous log formats, extract business metrics, and provide configurable, dynamic data processing for large‑scale logging scenarios.
DroolsElasticsearchFlink
0 likes · 6 min read

This article shares concise, easy-to-understand shell command techniques—using grep, sed, awk, tr, and vi—to efficiently delete empty lines from log files, highlighting each tool's syntax, options, and practical usage for embedded software developers.
Viawkgrep
0 likes · 5 min read

Facing massive, heterogeneous logs from numerous payment services, Ctrip’s data team designed a unified logging framework that extends log4j2, streams logs via Kafka to HDFS using a customized Camus pipeline, partitions and stores data in ORC for efficient Hive analysis, while addressing format, storage, and performance challenges.
CamusHadoopHive
0 likes · 16 min read

This article explains how Linux command‑line pipelines work, highlights the limitations of traditional shell scripts, and demonstrates how a concise Python script can replace them for counting unique users in log files while remaining easy to read, reusable, and powerful.
LinuxPythonlog processing
0 likes · 9 min read

The article describes the design, architecture, and operational details of a unified logging framework at Ctrip's payment center, covering log production via a Log4j2 extension, Kafka‑Camus collection, Hive/ORC storage, MapReduce parsing optimizations, and governance strategies for massive daily TB‑scale data.
CamusHadoopMapReduce
0 likes · 15 min read

This article explains the role of message‑queue middleware in distributed systems, covering its core functions, common use cases such as asynchronous processing, application decoupling, traffic shaping, log handling and messaging, and provides concrete architectural examples illustrating performance improvements and design patterns.
Asynchronous Processingapplication decouplingbackend architecture
0 likes · 9 min read

This article details Ctrip's migration of large‑scale log processing from Elasticsearch to ClickHouse, explaining why ClickHouse was chosen, the high‑availability deployment architecture, data ingestion strategies, dashboard integration, performance gains, operational practices, and overall cost and reliability improvements.
ClickHouseDatabase MigrationElasticsearch
0 likes · 12 min read

This article presents a Python script that reads performance test log files, extracts timing data, and visualizes the results as violin charts using Plotly, providing a reusable utility for server performance analysis and reporting.
data visualizationlog processingviolin chart
0 likes · 4 min read

The article explains how DBus ingests data from relational databases and log sources by detailing its extractor, incremental conversion, and full‑pull modules, the use of Canal and Kafka, rule‑based log structuring, the unified UMS message format, and heartbeat monitoring for reliability.
CanalDBusData Ingestion
0 likes · 13 min read

The guide explains how comprehensive data collection in big‑data platforms relies on a standardized event model, passive and code‑based embedding, multi‑platform SDKs, a log‑middleware layer, precise location tracking, and an embedding management platform that supports workflow, testing, quality monitoring, and scalable infrastructure for future enhancements.
AnalyticsSDKbig data
0 likes · 19 min read

This article describes a real‑time log‑classification pipeline built on Flink and Kafka that pre‑filters, structures, classifies, and aggregates heterogeneous logs, enabling efficient frequency‑based alerts and statistical analysis without storing raw log data at scale.
FlinkKafkaaggregation
0 likes · 11 min read

This article introduces message queue middleware, explains its role in distributed systems for asynchronous processing, system decoupling, traffic shaping, log handling and message communication, and provides concrete e‑commerce and log‑collection examples illustrating how queues improve performance, scalability and reliability.
Asynchronous ProcessingMessage QueueSystem Decoupling
0 likes · 8 min read

This article explains how message queue middleware improves distributed systems by enabling asynchronous processing, decoupling services, handling traffic spikes, aggregating logs, and supporting various communication patterns, illustrated with real‑world e‑commerce and logging architectures that boost throughput and reliability.
Asynchronous Processingapplication decouplinglog processing
0 likes · 9 min read

This article explains how the open‑source DBus platform, combined with the Wormhole streaming engine, captures raw application logs, lets users configure visual rule operators, and transforms the unstructured message part into schema‑driven, Kafka‑ready data for downstream analytics.
DBusbig datalog processing
0 likes · 15 min read

This article explains how Alibaba’s Sunfire monitoring platform processes terabytes of logs per minute, uses a pull‑based architecture with Brain‑Reduce‑Map roles, tackles scalability and reliability challenges, and outlines future directions such as MQL standardization and intelligent baselines.
OperationsReal-timelarge-scale
0 likes · 17 min read


21CTOAug 5, 2017 · Backend Development
This article walks through solving a massive log‑keyword counting task—600,000 short messages and 50,000 keywords—by evolving from a simple grep‑based approach to regex optimizations, word‑splitting, a trie data structure, and finally a multi‑process Redis queue, achieving a performance boost from hours to under ten minutes.
Triegreplog processing
0 likes · 15 min read

The Sentinel system was built to provide real‑time, zero‑modification monitoring of airline ticket business services by consuming Tianwang logs through a Storm cluster, offering flexible rule configuration, addressing performance pitfalls, and planning future enhancements such as custom monitoring scripts and visual dashboards.
KafkaReal-timeStorm
0 likes · 6 min read

This tutorial introduces the jq command‑line tool, shows how to download and install it, and demonstrates practical commands for extracting specific JSON fields from log data, chaining jq pipelines, and monitoring click and PV metrics in real time.
jqlog processing
0 likes · 6 min read

This article explores the evolution of log processing in the big‑data era, detailing how ELK Stack, Kafka, and Spark Streaming work together to provide scalable, real‑time log collection, analysis, and visualization for modern cloud‑native operations.
KafkaOperationsSpark
0 likes · 12 min read

CallGraph, JD.com’s in‑house distributed tracing platform, provides low‑intrusion, high‑performance monitoring for micro‑service ecosystems, enabling real‑time call‑graph analysis, TP metrics, flexible configuration, and future extensions such as deep‑learning‑driven insights.
Distributed Tracinglog processingmonitoring
0 likes · 15 min read

This article presents a detailed case study of Vipshop's log platform during the 419 sales event, analyzing the 2013 architecture, bottlenecks in RabbitMQ and Storm, and the subsequent redesign using Kafka, Impala, and HBase to achieve scalable, reliable big‑data processing.
ImpalaKafkaStorm
0 likes · 16 min read

The article analyzes Vipshop’s 419 flash‑sale log platform, detailing the 2013 architecture using Flume, RabbitMQ, Storm, Redis and MySQL, diagnosing bottlenecks in RabbitMQ and Storm during traffic spikes, and presenting practical scaling and monitoring solutions for high‑throughput backend systems.
RabbitMQStormlog processing
0 likes · 8 min read

This article explains Kafka's high‑throughput distributed messaging architecture, its typical scenarios such as log and operational data collection, the roles of producers, brokers, topics, consumers, Zookeeper, and provides a practical example of detecting abnormal user transactions.
Data StreamingKafkalog processing
0 likes · 3 min read

Learn how to handle high‑concurrency log collection by buffering logs in a Redis queue, then periodically batch‑inserting them into a database, with code examples for enqueuing, processing, error rollback, and scheduling the task.
Message Queuebatch insertlog processing
0 likes · 3 min read


21CTONov 4, 2015 · Big Data
To monitor hundreds of web apps on UAE’s PaaS platform in near‑real time, we combined Storm with lightweight log transport, a memcached‑based fqueue, and adaptive cardinality counting to efficiently compute PV, UV, response times, and custom metrics while handling cross‑cluster log aggregation.
Cardinality countingStormUAE platform
0 likes · 9 min read

This tutorial walks beginners through essential AWK commands and techniques for parsing and filtering log files, covering field extraction, separators, arithmetic on string fields, BEGIN/END blocks, conditional filters, external parameters, and common functions with practical examples.
awklog processingsysadmin
0 likes · 11 min read
