How to Set Up No‑Code Data Dump from Alibaba Cloud Kafka to OSS

Overview

Alibaba Cloud Message Queue for Kafka (MQ Kafka) offers a fully managed, serverless sink connector that streams records from a Kafka topic directly to an Object Storage Service (OSS) bucket without writing custom integration code.

Key Benefits

Simple : Only configuration is required; no development effort.

Fully Managed : Serverless compute removes operational overhead.

Cost‑Effective : You pay only for Function Compute usage; no extra connector fees.

Typical Use Cases

Backup and archival of Kafka data to OSS for disaster recovery.

Big‑data analytics by using OSS as a durable storage layer for logs, sensor data, or user‑behavior files.

Caching intermediate results of Function Compute (FC) jobs in OSS.

Connector Capabilities

Zero‑code development: a built‑in connector handles data transfer.

Configurable routing and storage policies (by time, keyword, topic, folder, filename, etc.).

High reliability and fault tolerance with automatic redundancy handling and failure recovery.

Serverless scaling: resources expand or shrink automatically based on load.

Prerequisites

A running MQ Kafka instance in the desired region.

Appropriate IAM permissions for creating connectors and accessing OSS.

An OSS bucket (e.g., oss-sink-connector-bucket) created in the OSS console.

Step‑by‑Step Setup

Log in to the MQ Kafka console and select the target region.

In the left navigation, choose Connector → Sink and click Create Task .

In the task creation panel, configure:

Task name.

Sink type: OSS .

Source topic (the Kafka topic to export).

Target bucket name.

File format (e.g., text or json).

Partition handling options (e.g., one file per partition, size‑based rolling).

Optional routing rules (time‑window, keyword filtering, custom folder structure).

Confirm the configuration. The task status changes from Starting to Running once the connector is created.

Test the connector: produce messages to the source topic and verify that files appear in the OSS bucket.

Verification

Two kinds of objects are created in the bucket:

System meta files : named .oss_meta_file_partition_{partitionID}, one per upstream partition, used internally for batch tracking.

Data files : named partition_{partitionID}_offset_{offset}_{random}. Each file may contain multiple messages from the same partition; the offset part is the smallest Kafka offset in that file. Messages are separated by newline characters.

Download the objects via the OSS console, open them with a text editor, and confirm that the message payloads match the produced records.

References

Kafka instance preparation: https://help.aliyun.com/zh/apsaramq-for-kafka/getting-started/getting-started-overview

Prerequisites for connectors: https://help.aliyun.com/zh/apsaramq-for-kafka/user-guide/prerequisites#concept-2323967

Create OSS bucket: https://help.aliyun.com/zh/oss/getting-started/console-quick-start#task-u3p-3n4-tdb

MQ Kafka console login: https://account.aliyun.com/login/login.htm?oauth_callback=https%3A%2F%2Fkafka.console.aliyun.com%2F%3Fspm%3Da2c4g.11186623.2.22.6bf72638IfKzDm&lang=zh