How 360’s Qcmd‑HTTP Powers Distributed Task Execution on the Cloud

Introduction

Qcmd‑HTTP is a task distribution layer built in Golang on top of the 360 cloud platform’s underlying Qcmd command execution system. It provides cluster‑level distribution, high availability, support for complex task types, and complete decoupling of business logic from the low‑level Qcmd engine.

Core Functions

Implement distributed clustering and maintain high availability of Qcmd.

Support more complex task types to meet diverse business scenarios.

Fully decouple business services from the underlying Qcmd.

Execution Process

From the platform UI, users select a script, choose target hosts (minions), and launch the task. The task status progresses from processing to success , and users can view real‑time status and per‑minion results at any time.

Note: The zjl.sh script simply prints the current hostname.

Terminology

Minion : The host that actually executes the command, e.g., hostname .

Master : Receives business task requests, dispatches them to minions, collects results, and reports back to the requester.

Cluster : A set of master nodes (typically two for high availability) together with their associated minions.

Distributed : Multiple clusters are deployed across data centers to reduce latency and balance load, forming a distributed topology.

Distributed Cluster Topology

Cluster Status

Master status : Masters operate in a primary‑secondary configuration; the system automatically promotes a secondary to primary if the primary fails.

Minion status : Minions report up or down via periodic heartbeat; a down minion is marked accordingly in the database.

Task Types

Single task – serial blocking: Executes on multiple minions sequentially; stops on first failure.

Single task – serial non‑blocking: Continues to the next minion even if a previous one fails.

Single task – parallel: Executes the same command on all selected minions concurrently.

Group task – serial: A complex workflow (e.g., MySQL instance creation) composed of multiple sub‑tasks that run sequentially, aborting on any failure.

Task Status

Host execution status: Minions transition from waiting to processing to success or failure as reported by the master.

Single task status: Aggregated from the statuses of its constituent minion executions.

Group task status: Derived from the statuses of all sub‑tasks, with overall states such as waiting, processing, success, failure, or timeout.

Callback Mechanism

All tasks run asynchronously. Upon submission, a job_id (for single tasks) or group_id (for group tasks) is returned. The system provides query APIs to retrieve results, and optionally accepts a callback URL that is invoked with the final outcome.

Overall Design

Conclusion

The primary goal of Qcmd‑HTTP is to fully decouple business services from the low‑level Qcmd engine, resulting in a clearer architecture and well‑defined responsibilities. Since its launch, it has served for over a year, executing more than 500,000 single tasks and 200,000 group tasks.