Enhancements in ACK One Serverless Argo Workflows: Large File Upload, Artifact GC, and Streaming Transfer

Name Explanation

Argo Workflows: a cloud‑native workflow engine designed to coordinate multiple tasks on a Kubernetes cluster.

Artifacts: persistent data objects produced during workflow execution, typically representing step outputs.

OSS: Alibaba Cloud Object Storage Service, commonly used to store Artifacts.

Background

Argo Workflows is an open‑source CNCF graduated project that automates complex workflows on Kubernetes, suitable for scheduled jobs, machine learning, ETL, model training, data pipelines, CI/CD, etc.

When using Argo Workflows for data‑intensive tasks such as model training or bioinformatics analysis, efficient management of large Artifacts stored in OSS becomes critical. Open‑source solutions face challenges:

Uploading files larger than 5 GiB fails due to client limits.

Lack of automatic file cleanup leads to unnecessary OSS storage consumption.

Argo Server downloads files to local disk first, causing high disk usage and potential service disruption.

ACK One Serverless Argo Workflows addresses these issues with three major enhancements.

1. Support for Chunked Large‑File Upload

The service adds multipart, resumable upload to OSS, enabling files larger than 5 GiB to be uploaded reliably. Each chunk includes integrity checks, improving data safety and fault tolerance.

Usage Example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artifact-
spec:
entrypoint: main
templates:
- name: main
metadata:
annotations:
k8s.aliyun.com/eci-extra-ephemeral-storage: "20Gi"  # custom temporary storage
k8s.aliyun.com/eci-use-specs : "ecs.g7.xlarge"
container:
image: alpine:latest
command:
- sh
- -c
args:
- |
mkdir -p /out
dd if=/dev/random of=/out/testfile.txt bs=20M count=1024 # generate 20 Gi file
echo "created files!"
outputs: # trigger upload to OSS
artifacts:
- name: out
path: /out/testfile.txt

2. Support for OSS Artifacts Garbage Collection (GC)

The service introduces configurable GC strategies to automatically delete Artifacts after workflow completion or deletion, reducing storage costs and preventing unlimited growth.

Usage Example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artifact-gc-
spec:
entrypoint: main
artifactGC:
strategy: OnWorkflowDeletion # global policy, can be overridden
templates:
- name: main
container:
image: argoproj/argosay:v2
command:
- sh
- -c
args:
- |
echo "hello world" > /tmp/on-completion.txt
echo "hello world" > /tmp/on-deletion.txt
outputs: # upload files to OSS
artifacts:
- name: on-completion
path: /tmp/on-completion.txt
artifactGC:
strategy: OnWorkflowCompletion # override global policy
- name: on-deletion
path: /tmp/on-deletion.txt

3. Support for Streaming Transfer of Artifacts

Instead of downloading files to the Argo Server’s local disk, the service streams files directly from OSS to the client via the OpenStream interface, reducing latency, disk usage, and improving concurrency and security.

Summary

ACK One Serverless Argo Workflows, as a fully managed service, offers the following advantages over the open‑source version:

OSS File Management Capability

Open‑source Argo Workflows

ACK One Serverless Argo

File Upload

Supports only <5 Gi files

Supports chunked large‑file upload

File Cleanup

Not supported

Supports Artifact GC

File Download

Requires local staging

Supports streaming transfer

Future releases will contribute these enhancements back to the community, continuing to improve stability, usability, and performance of the serverless Argo platform.