How to Build a Low‑Cost, High‑Throughput Serverless Message Consumer on Alibaba Cloud

Background

Message Service is a core PaaS component in cloud computing, offering high concurrency and peak‑shaving capabilities. Developers often need to transport messages from Message Service to downstream consumers efficiently, while also building a serverless, elastic, and maintenance‑free consumption service.

Key Cloud‑Native Components

Function Compute : An event‑driven, fully managed serverless compute service. Users upload code without managing servers; the platform automatically provisions resources with elastic scaling.

Connector : Provides data import/export between systems (e.g., Kafka, RocketMQ) and simplifies data replication.

EventBridge : A centralized event bus that routes events between Alibaba Cloud services, SaaS applications, and custom apps.

Typical Consumption Architecture and Pain Points

Traditional flow: data source → Message Service → OpenAPI/SDK/Proxy → consumer logic → target service. This leads to issues such as concurrent‑safe reads, limited consumption throughput, scaling bottlenecks, real‑time guarantees, fault tolerance, monitoring, and cost control.

Connector Solution Overview

Connector bridges Message Service and Serverless Function Compute, offering a declarative way to define upstream message sources and downstream processing operators. It provides the following capabilities:

Transform : Custom data cleaning via UDF or JsonPath extraction.

Filter : Drop irrelevant messages using prefix/suffix, numeric, or IP matching.

Window : Batch messages by count or time interval to improve throughput and reduce cost.

Real‑Time : Millisecond‑level latency when window interval is set to zero.

Custom Concurrency : Configure multiple consumer threads for parallel partition consumption.

Elastic Compute : Function Compute auto‑scales based on load.

Monitoring + Logging + Tracing : Rich metrics and logs for observability.

Robust Fault‑Tolerance : Retry policies, dead‑letter queues, rate limiting, and back‑pressure mechanisms.

Detailed Feature Benefits

Window

Two configurable parameters:

Batch push count – maximum number of messages aggregated before forwarding.

Batch push interval – time interval for aggregation; zero disables waiting.

Use cases include real‑time streaming (interval = 0, count = max) and low‑frequency batch processing (adjust count and interval to avoid excessive latency).

Transform

Data cleaning can be performed via:

Template : Simple JsonPath‑based extraction for fixed‑structure data.

UDF : User‑defined Python function for complex transformations.

# -*- coding: utf-8 -*-</code><code>def handle_message(event, context):</code><code>    try:</code><code>        new_message = transform(event)</code><code>    except Exception as e:</code><code>        raise e</code><code>    return new_message</code><code></code><code>def transform(old_message):</code><code>    # custom cleaning logic</code><code>    return new_message

Filter

Supports multiple matching modes (prefix, suffix, numeric, IP) to discard unwanted messages, reducing downstream processing and function invocations.

Real‑Time Mode

Setting the window interval to zero turns batch processing into real‑time consumption, achieving millisecond‑level end‑to‑end latency.

Custom Concurrency

For Kafka, the number of consumer threads can be matched to partition count, enabling parallel consumption while preserving order per partition.

High‑Availability Strategies

Retry policies with exponential back‑off.

Dead‑letter queues (Kafka, RocketMQ, MNS) for failed messages.

Fault‑tolerance modes: allow or forbid continuation after errors.

Back‑pressure control for upstream pull speed and downstream rate limiting.

Connector Architecture

Connector defines three task types:

Poller Task – pulls messages from Message Service.

Transform Task – applies cleaning, filtering, aggregation.

Sink Task – pushes processed messages to the target service.

All tasks are horizontally scalable, allowing concurrent consumption of multiple partitions and parallel delivery.

Customer Use Case

A advertising platform streams user‑view logs into Kafka with heterogeneous formats. Requirements: data cleaning, low cost, and scalability. The solution uses Function Compute with Connector to:

Implement custom cleaning via UDF in Transform Task.

Reduce cost by aggregating messages with Window and discarding noise with Filter.

Achieve horizontal scalability by adjusting partition count, poller instances, sink workers, and Function Compute quotas.

Step‑by‑Step Implementation

Create a Kafka instance, topic, and consumer group via the Kafka console.

Create a Function Compute service and a function, then configure a Kafka trigger with the created resources.

Optionally enable batch push (e.g., batch count = 2, interval = 10 s) to test windowing.

Deploy the function.

Write the function handler to log received messages:

# -*- coding: utf-8 -*-</code><code>import logging, json</code><code>def handler(event, context):</code><code>    msgs = json.loads(event)</code><code>    logging.getLogger().info(len(msgs))  # number of messages</code><code>    logging.getLogger().info(msgs)      # content</code><code>    return 'succ'

Send test messages to the Kafka topic and observe that the function is invoked according to the batch settings.

Summary & Outlook

Using Serverless Function Compute with Connector provides a secure, reliable, and cost‑effective data‑consumption system. Benefits include:

Cost reduction : Filter and Window lower function invocations; pay‑as‑you‑go pricing avoids idle resources.

Efficiency gains : Built‑in Transform, UDF, and Template accelerate development; rich observability improves debugging and operations.

Scalability : All components (Kafka partitions, pollers, sink workers, compute instances) can be scaled horizontally to match traffic growth.

As cloud computing moves toward full serverless adoption, tighter integration between Message Service and Function Compute will further lower the barrier for building end‑to‑end data pipelines.