How Serverless Functions Achieve Exactly‑Once Task Deduplication

Preface

In both big‑data and message‑processing domains, task systems must provide task‑trigger deduplication, especially in high‑accuracy scenarios such as finance. Serverless Task, as a serverless task‑processing platform, needs to guarantee accurate trigger semantics at both the user‑application and system‑internal levels.

Brief Discussion on Task Deduplication

Message processing in an asynchronous task system follows a simple flow: user submits a task, it enters a queue, a task‑processing unit listens and fetches the message, then schedules it to a worker for execution. Any component failure (e.g., crash) can cause incorrect message delivery. Typical systems offer up to three delivery semantics:

At‑Most‑Once – a message is delivered at most once; failures may cause loss.

At‑Least‑Once – a message is delivered at least once; retries may cause duplicates.

Exactly‑Once – a message is delivered exactly once; achieved by combining retries with idempotent processing (deduplication).

Function Compute provides Exactly‑Once semantics for task dispatch: duplicate tasks are recognized as the same trigger and dispatched only once.

To achieve deduplication, the system must ensure two dimensions of protection:

System‑side guarantee: the scheduler’s failover does not affect message uniqueness.

User‑side guarantee: the application can enforce trigger‑plus‑execution deduplication based on business logic.

Implementation Background of Async Task Deduplication in Function Compute

The architecture consists of an API‑Server, internal queue, asynchronous listener, resource manager, scheduler, and VM‑level client. Users submit a task via the Function Compute API (step 1), which is validated and placed into the internal queue (step 2.1). An asynchronous module listens to the queue (step 2.2), obtains runtime resources (step 2.2‑2.3), and the scheduler dispatches the task to a VM‑level client (step 3.1) which forwards it to the actual execution environment (step 3.2).

System‑side guarantee: any failover between steps 2.1‑3.1 triggers the task only once. User‑side guarantee: repeated submissions of step 1 result in only one execution of step 3.2.

When a message enters the system, the API returns HTTP 202, indicating successful submission. The scheduler and underlying RocketMQ queue provide three‑replica storage and unique consumption semantics. Each message becomes invisible after consumption, preventing other consumers from retrieving it.

Scheduler responsibilities include:

Listening to its assigned queue according to load‑balancing policies.

Pulling messages, maintaining an in‑memory state, and continuously extending the message’s visibility timeout until the user instance returns a result.

Deleting the message after successful execution.

During scheduler failover or upgrade, two cases arise:

If the message has not yet been dispatched to the user instance, another scheduler will pick it up after the visibility timeout and trigger execution.

If the message is already executing, the user‑side agent detects the duplicate request and ignores it, reporting the result back to the new scheduler.

User‑Side Business‑Level Deduplication

To handle repeated submissions caused by network partitions, Function Compute introduces a globally unique TaskID (StatefulAsyncInvocationID). The same TaskID submitted multiple times results in a 400 error, allowing the client to know the task was already accepted.

import fc "github.com/aliyun/fc-go-sdk"

func SubmitJob() {
    invokeInput := fc.NewInvokeFunctionInput("ServiceName", "FunctionName")
    invokeInput = invokeInput.WithAsyncInvocation().WithStatefulAsyncInvocationID("TaskUUID")
    invokeOutput, err := fcClient.InvokeFunction(invokeInput)
    // handle output and error
}

Conclusion

The article explains how Serverless Task achieves exactly‑once task triggering, ensuring strict execution accuracy for scenarios that demand it. By leveraging system‑side failover protection and user‑side unique TaskID deduplication, each submitted task is executed precisely once.