Boost Data Analysis and ETL with Alibaba Cloud Function Compute Async Tasks

Background

Offline workloads such as data processing, machine‑learning training, and statistical analysis often require long execution times and high concurrency. Python is the dominant language for these tasks, and Alibaba Cloud Function Compute provides a native Python runtime with easy third‑party library integration, making asynchronous task execution convenient and cost‑effective.

Typical Requirements for Data‑Analysis Jobs

Developer‑friendly environment that supports third‑party packages and custom dependencies.

Ability to run long‑running jobs, monitor task status, and manually stop tasks on error.

High resource utilization and optimal cost.

Case Study – Database Autonomous Service

The internal database inspection platform processes slow‑query logs and other SQL metrics. Online analysis can consume tens of thousands of cores, while offline training uses tens of thousands of core‑hours per day. A Flink‑based solution suffered from:

Complex deployment, testing, and release cycles.

Poor support for common third‑party Python libraries.

Lengthy debugging paths.

Insufficient elasticity and high cost during peak periods.

By migrating core training and statistical algorithms to Function Compute asynchronous tasks, the platform achieved:

Full handling of peak traffic and timely daily analysis.

Rapid iteration thanks to rich Function Compute runtime capabilities.

Compute cost reduced to roughly one‑third of the original Flink solution.

Developers were freed from platform‑operations concerns and could focus on algorithm development.

Best Practice – Kafka ETL

ETL jobs typically consist of a source, a sink, and processing logic. They require strong upstream/downstream connectivity, exactly‑once semantics, deduplication, and failure compensation. Function Compute asynchronous tasks provide:

Configurable success and failure destinations that automatically route results or dead‑letter messages.

Support for custom operators and third‑party Python libraries via packaged runtimes.

Kafka ETL Example Steps

Create a Kafka instance and a test topic.

Create two MNS queues: dead-letter-queue (for failed messages) and fc-etl-processed-message (for successful results).

Install Serverless Devs: npm install @serverless-devs/s Add credentials: s config add Modify s.yaml to set the target MNS ARN and service role, then deploy: s deploy -t s.yaml In the Kafka console, create a connector, select Function Compute as the destination, and enable asynchronous mode.

After deployment, each Kafka message triggers the function. The function returns a payload with requestPayload (original Kafka event) and responsePayload (function output). Successful and failed messages are automatically delivered to the configured MNS queues.

Best Practice – Audio‑Video Processing

Video‑on‑demand solutions need massive storage, transcoding, CDN acceleration, and content‑security checks. Serverless asynchronous tasks provide:

Pay‑as‑you‑go compute with automatic scaling.

Support for up to 24‑hour long‑running jobs.

Task deduplication and exactly‑once processing.

Full observability of task lifecycle, logs, and metrics.

Ability to stop or retry tasks on demand.

Fast development and testing via Serverless Devs tools.

FFmpeg Video Transcode Example

Install Serverless Devs: npm install @serverless-devs/s Configure credentials: s config add Initialize the project: s init video-transcode -d video-transcode Deploy the function: cd video-transcode && s deploy Invoke the transcode function asynchronously, for example:

$ s VideoTranscoder invoke -e '{"bucket":"my-bucket", "object":"480P.mp4", "output_dir":"a", "dst_format":"mov"}' --invocation-type async --stateful-async-invocation-id my1-480P-mp4

The FC console shows task status, start/end times, logs, and payloads. If a task fails, a dest-fail function can be configured for custom alerting.

References

Function Compute async task source code: https://github.com/awesome-fc/Stateful-Async-Invocation

Serverless Devs installation guide: https://github.com/Serverless-Devs/Serverless-Devs/blob/master/docs/zh/install.md

Alibaba Cloud key configuration documentation: https://github.com/devsapp/fc/blob/main/docs/zh/config.md

Content security documentation: https://help.aliyun.com/product/28415.html

FFmpeg transcode project source: https://github.com/devsapp/start-ffmpeg/tree/master/transcode/src

FC console overview: https://fcnext.console.aliyun.com/overview