How ACK One Serverless Argo Solves Large‑File Artifact Challenges in Cloud‑Native Workflows

Background

Argo Workflows is a CNCF‑graduated, cloud‑native workflow engine that orchestrates containerized tasks on Kubernetes. While it simplifies complex pipelines such as CI/CD, ETL, and machine‑learning jobs, the open‑source version lacks robust support for large artifact files, leading to upload failures, storage bloat, and high disk usage on the Argo Server.

Key Challenges

Multipart upload limit: Files larger than 5 GiB cannot be uploaded because the client enforces a size cap.

No automatic artifact cleanup: Temporary files and completed task outputs remain in OSS, consuming storage indefinitely.

Argo Server disk pressure: Downloading artifacts requires writing them to the server’s local disk first, which can exhaust disk space and affect stability.

ACK One Serverless Argo Enhancements

1. Multipart Upload & Resume Support

ACK One introduces a chunked upload mechanism for OSS that splits large files into independent parts, each verified for integrity. This enables reliable transfer of files >5 GiB, reduces the risk of network interruptions, and improves overall throughput for data‑intensive workloads.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-
spec:
  entrypoint: main
  templates:
  - name: main
    metadata:
      annotations:
        k8s.aliyun.com/eci-extra-ephemeral-storage: "20Gi"
        k8s.aliyun.com/eci-use-specs: "ecs.g7.xlarge"
    container:
      image: alpine:latest
      command: ["sh", "-c"]
      args:
      - |
        mkdir -p /out
        dd if=/dev/random of=/out/testfile.txt bs=20M count=1024 # generate 20 Gi file
        echo "created files!"
    outputs:
      artifacts:
      - name: out
        path: /out/testfile.txt

2. Automatic Artifact Garbage Collection (GC)

The platform adds a configurable artifactGC policy that can delete artifacts after workflow completion or after the workflow object is removed. Users can also leverage OSS lifecycle rules to purge old files or archive them to cold storage.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-gc-
spec:
  entrypoint: main
  artifactGC:
    strategy: OnWorkflowDeletion
  templates:
  - name: main
    container:
      image: argoproj/argosay:v2
      command: ["sh", "-c"]
      args:
      - |
        echo "hello world" > /tmp/on-completion.txt
        echo "hello world" > /tmp/on-deletion.txt
    outputs:
      artifacts:
      - name: on-completion
        path: /tmp/on-completion.txt
        artifactGC:
          strategy: OnWorkflowCompletion
      - name: on-deletion
        path: /tmp/on-deletion.txt

3. OSS Streaming Download (OpenStream)

Instead of writing files to the Argo Server’s local disk, ACK One streams artifacts directly from OSS to the client. This reduces latency, lowers memory and disk consumption, improves concurrency, and minimizes security exposure of temporary data.

Faster downloads: Users receive the first byte as soon as the stream starts.

Lower resource usage: Server disk and RAM remain largely untouched, allowing more parallel transfers.

Enhanced compliance: No intermediate storage on the server reduces the risk of data leakage.

Conclusion

By integrating multipart uploads, automatic garbage collection, and OSS streaming, ACK One Serverless Argo addresses the three major pain points of the open‑source Argo Workflows—large‑file handling, storage cleanup, and server disk pressure—providing a more efficient, secure, and scalable solution for cloud‑native data‑intensive pipelines.