How Hulk’s Private Cloud Optimizes SaltStack for Scalable Command Execution

Overview

SaltStack is a widely used cluster management solution. This article describes how the Hulk private‑cloud platform customizes and optimizes SaltStack for large‑scale command execution and explains key internal mechanisms.

Native SaltStack Architecture

SaltStack consists of a master and multiple minions. The master and minions keep a persistent TCP connection encrypted with AES. Commands are dispatched using ZeroMQ’s publish‑subscribe pattern; the master publishes, and each minion receives, executes, and returns results. While manual execution works for a few machines, it becomes impractical for thousands.

Hulk’s Customized Command System

The system builds on the native SaltStack structure and adds three functional layers:

Integrated command module in the Hulk management console : manages predefined task scripts, displays execution status and results, supports retry of failed tasks, and records execution history.

Custom task module : pulls execution results, reports the controllable status of each minion host.

Native salt‑master and salt‑minion plus custom Perl scripts : the Perl scripts verify that task scripts are up‑to‑date, update them if necessary, monitor execution timeouts, and terminate overdue processes.

Data Flow with Redis

To decouple task creation from execution, Redis is used between the first two layers. Tasks and target hosts are written to Redis; the second layer retrieves them and invokes the third layer.

Execution Workflow (Steps 1‑7)

Hulk’s command system validates selected hosts and checks their controllable status.

The system queries Redis for the SaltStack cluster of each host and pushes the host‑command pair into the cluster‑specific Redis queue.

Each cluster’s task module pulls messages from its queue, calls the salt‑master, and marks the task as “in progress”.

The master signs the message with its private key and publishes it via ZeroMQ to all minions.

Minions decrypt the message with the master’s public key, verify the host ID, and, if it matches, spawn a subprocess to run the command.

After execution, the minion returns the result to the master; the task control module writes each host’s result to Redis. If a timeout occurs, the Perl script kills the process and the system marks the task as timed‑out.

The command system aggregates results from Redis, merging hosts with identical outcomes.

Current Performance and Future Work

The current system operates online with a 99 % success rate, but limitations have been identified: ZeroMQ’s pub/sub can be unreliable, broadcasting commands to all minions wastes network bandwidth, and failure/timeout handling is insufficient. Hulk is developing a new command system to address these issues.