How to Seamlessly Import Massive S3 Logs into Alibaba Cloud SLS with Real‑Time Analysis

Background

Enterprises that run workloads across multiple clouds generate large volumes of log files in object storage services such as AWS S3. To achieve centralized operations, security compliance and unified analytics, these scattered logs must be imported into a single log platform – Alibaba Cloud Log Service (SLS) – for real‑time querying and alerting.

Technical Challenges

Massive small‑file discovery: Services like CloudTrail or ALB write thousands of tiny objects per minute. S3’s ListObjects API only supports lexicographic traversal, so finding the newest files may require scanning the entire bucket, which is too slow for real‑time ingestion.

Elastic response to traffic spikes: E‑commerce promotions or system failures can cause sudden log‑volume surges. The import pipeline must detect the change and scale within minutes.

Diverse data formats and cost control: Logs may be compressed (gzip, snappy, lz4, zstd, zip, none), stored as JSON, CSV, Parquet, plain text, CloudTrail, etc., and often contain dirty data that requires field extraction, conversion or masking. Importing raw data first then processing it incurs extra storage and compute cost.

Solution Overview – Two‑Stage Parallel Architecture

Stage 1 – File discovery: Combines real‑time event capture with periodic full scans to guarantee “no‑miss”.

Stage 2 – Data pulling: A dedicated transfer channel pulls files independently of the scanning process, ensuring high throughput.

Key benefit: Both stages run in parallel, delivering speed and reliability.

File Discovery Modes

Full‑scan mode: Periodically (e.g., every minute) scans the whole directory to ensure completeness for strict integrity requirements. Imported files are recorded to avoid duplicates.

Incremental mode: Uses lexicographic ordering to resume from the last scanned position, quickly discovering newly added files. Works best when file names are time‑ordered, achieving minute‑level latency.

Combined use: Incremental mode provides timeliness; full‑scan acts as a safety net for completeness.

SQS Event‑Driven Import

Configure S3 event notifications to send a message to an SQS queue whenever a new object is created.

The import service consumes messages from SQS and imports the specified file directly, bypassing any scan.

This yields sub‑minute latency, ideal for unordered file creation or strict real‑time alerting.

Elastic Scaling Mechanisms

Sliding‑window adaptive adjustment: Every 5 minutes the system evaluates pending data volume, estimates required concurrency based on file size/quantity and automatically scales up or down.

Long‑tail optimization: Balances task workloads so that a few heavy files do not delay the overall import.

User‑specified concurrency: Users can pre‑set the maximum concurrency for anticipated traffic peaks via a ticket.

Pre‑Ingestion Data Processing (Cost‑Effective)

Processing is performed before data is written to SLS, eliminating the “store‑then‑process” overhead. Supported operations include:

Field extraction: Pull key fields from unstructured logs.

Data filtering: Discard irrelevant logs.

Field conversion: Standardize formats, convert timestamps.

Data masking: Anonymize sensitive information.

Example processor rule:

* | parse-csv -delim='\t' content as time,level,order_id,amount,currency,error_code,response_time,status_code,client_id,customer_email,id_card
| project-away content
| extend customer_email = regexp_replace(customer_email, '([\s\S]+)@([\s\S]+)', '****@\2')
| extend id_card = regexp_replace(id_card, '(\d{3})(\d+)(\d{3})', '\1*****\3')
| extend __time__ = cast(to_unixtime(cast(time as TIMESTAMP)) as bigint) - 28800

Supported Formats

Compression: zip, gzip, snappy, lz4, zstd, none

Data format: JSON, CSV, single‑line text, multi‑line text, CloudTrail, JSON array, etc.

Character encoding: UTF‑8, GBK

Best‑Practice Scenarios

Scenario 1 – AWS Service Logs (Dual‑Mode Traversal)

Typical logs: CloudTrail, VPC Flow Logs, S3 access logs with monotonically increasing filenames.

Configuration: enable both incremental and full‑scan modes, set a write processor for key‑field extraction, and check for new files every minute.

Result: end‑to‑end latency of 2‑3 minutes with 100 % data completeness.

Scenario 2 – Application Logs (SQS‑Driven Import)

Typical logs: Real‑time application logs with irregular file naming that require fast alerts.

Configuration: enable S3 event notifications to SQS and use the SQS‑driven import flow.

Result: sub‑2‑minute latency, satisfying strict real‑time alert requirements.

Performance & Reliability

Elastic scaling can automatically expand to >300 concurrent import tasks, achieving an import speed of ~5.8 GB/s in large‑scale tests.

File‑level status tracking, automatic retry, and completeness verification guarantee reliable ingestion.

Monitoring and alerting on import latency and failure rate are built‑in.

Cost Optimization

Pre‑write processing removes unnecessary data, reducing storage consumption.

Incremental import ensures only new or changed files are transferred, avoiding duplicate work.

Conclusion

The S3‑to‑SLS pipeline is more than a simple data mover; it is a fully engineered solution that addresses file‑discovery difficulty, traffic‑spike elasticity, format diversity and cost efficiency. By combining dual‑mode intelligent traversal, three elastic scaling mechanisms and pre‑write processing, the system delivers high‑performance, reliable and economical cross‑cloud log integration.