How to Seamlessly Integrate Multi‑Cloud Data with Alibaba Cloud SLS Object Import

Multi‑Cloud Data Fusion Challenges

Enterprises that adopt multiple cloud providers to avoid vendor lock‑in store business logs in different cloud file systems, making unified observation difficult. Key challenges include delayed detection of new files, lack of time‑ordered bucket traversal, the need for elastic scaling to handle periodic traffic spikes, and heterogeneous log schemas across clouds.

Why Choose SLS Object Import?

Alibaba Cloud Log Service (SLS) provides a cloud‑native observation platform with a high‑performance, easy‑to‑use object import feature that simplifies cross‑cloud data ingestion.

Object Import Architecture Overview

The architecture separates file discovery from data pulling and runs both stages in parallel, eliminating the discovery bottleneck and maximizing throughput.

File discovery uses diversified intelligent strategies to efficiently capture file changes while ensuring data completeness.

Data pulling operates independently, focusing on high‑speed data transfer.

Parallel execution of both stages maximizes performance.

Fast New‑File Detection

To discover new files within billions of objects quickly, SLS employs multiple strategies:

Periodic full bucket traversal for 100% coverage.

Incremental bucket traversal in lexicographic order to detect new files within a minute.

OSS metadata index assistance for OSS scenarios.

AWS SQS assistance for S3 scenarios.

These methods form a robust ecosystem that reliably identifies new files across cloud storage services.

Supported File and Compression Formats

SLS can parse single‑line text logs, multi‑line text logs, single‑line JSON, CSV, ORC, Parquet, etc. Supported compression formats include zip, gzip, zstd, lz4, snappy, bzip2, deflate.

Elastic Scaling for Traffic Spikes

SLS object import uses elastic scaling to handle sudden traffic increases without latency, automatically adjusting concurrency based on file count and size.

SLS Object Import Hands‑On

OSS Data Integration

Example: Import files created in April 2025 from an OSS bucket into the SLS logstore oss‑ingestion‑test .

Data Preparation

Ensure the data files are stored in any OSS bucket.

Create OSS Import Task

Configure permissions, select the target logstore, set the source region (e.g., Hangzhou), specify the bucket and prefix ingestion-test/2025/04, define data format (e.g., single‑line JSON with snappy compression), and disable periodic checks for one‑time historical import.

After previewing the import result, create the task.

View Task Status and Imported Data

Use the task management console to locate the task, view its overview, and inspect the imported data in the target logstore.

S3 Data Integration

Similar steps apply to Amazon S3. After granting AWS AccessKey ID and SecretAccessKey, optionally enable SQS for large buckets, set the source region (e.g., ap‑northeast‑1), and configure data and compression formats.

Preview and create the S3 import task.

Real‑World Use Cases

Cross‑Cloud Billing Audit

Export billing logs from Alibaba Cloud to OSS and from AWS to S3, then import them into SLS. Standardize fields with processing statements such as:

* | project-rename product=ProductCode | extend cost=PretaxGrossAmount | extend originProduct='aliyun' | project product, cost, originProduct

* | project-rename product=line_item_product_code | extend cost=pricing_public_on_demand_cost | extend originProduct='aws' | project product, cost, originProduct

Run SQL queries to aggregate daily costs per product across clouds.

* | SELECT SUM(cost) AS cost, date_trunc('day', dt) AS dt, originProduct FROM (SELECT SUM(CASE WHEN originProduct='aws' THEN cost*7.19 ELSE cost END) AS cost, date_parse(date, '%Y-%m-%d %H:%i:%S') AS dt, originProduct FROM log WHERE cost>0 GROUP BY dt, originProduct) GROUP BY dt, originProduct

Cross‑Cloud Operation Audit

Import operation audit logs from both clouds, normalize them, and analyze usage patterns. Example processing for AWS CloudTrail logs:

* | expand-values -path='$.Records' content as item | parse-json item | project-away item | extend originProduct='aws'

For Alibaba Cloud audit logs:

* | parse-json event | project-away event | extend originProduct='aliyun'

Sample queries include hourly operation counts per product and detailed event analysis.

* | SELECT COUNT(*) AS num, date_trunc('hour', __time__) AS dt, product FROM log GROUP BY dt, product LIMIT 100000000

* | SELECT COUNT(*) AS num, date_trunc('hour', __time__) AS dt, eventName FROM log WHERE product='Cms' GROUP BY dt, eventName LIMIT 100000000

Set alerts for critical events such as resource deletions:

* | SELECT originProduct, eventName, eventId, userAgent WHERE eventName LIKE '%Delete%'

Best Practices and Tips

Compress large files with zstd to reduce public‑network transfer costs.

When bucket file count exceeds one million, create new files in lexicographic order to ensure detection within two minutes.

Prefer many small files over few large ones to increase parallelism.

Organize logs by business‑specific prefixes and run multiple import tasks concurrently.

Always add new files instead of appending to existing ones to avoid duplicate data.

Conclusion

SLS object import provides a unified, high‑performance solution for multi‑cloud data ingestion, supporting OSS and S3 today and expanding to more providers in the future.