Parallelizing Bash Loops Without Extra Tools: Practical Shell Techniques

Background

Linux system administrators often need to modify configuration files or deploy applications across many hosts. A typical approach uses for or while loops in a shell script, which run serially and take * × iterations time, becoming a bottleneck for large‑scale tasks.

Why Parallel Execution?

Serial loops execute each iteration as a foreground process, allowing only one active child at a time. To run tasks concurrently, the loop body must be launched as a background process, letting the shell manage multiple child processes simultaneously.

Example 1 – Simple Background Execution

First, a serial script ( para-0.sh) demonstrates the baseline:

#!/bin/bash
Njob=15    # total tasks
for ((i=0; i<$Njob; i++)); do
{
  echo "progress $i is sleeping for 3 seconds zzz…"
  sleep 3
}
done
echo -e "time-consuming: $SECONDS seconds"    # show elapsed time

Running this script shows a total runtime of 45 seconds (15 × 3 s), confirming the serial behavior.

To parallelize, the loop body is sent to the background and the script waits for all children before reporting time:

#!/bin/bash
Njob=15
for ((i=0; i<$Njob; i++)); do
  echo "progress $i is sleeping for 3 seconds zzz…"
  sleep 3 &    # launch in background
done
wait            # wait for all background jobs
echo -e "time-consuming: $SECONDS seconds"

This version finishes in roughly 3 seconds, because all 15 sleep processes run concurrently. However, it lacks any control over the number of simultaneous processes, which can overwhelm system resources.

Example 2 – Simulated Queue to Limit Concurrency

To avoid resource exhaustion, a fixed‑size “queue” is simulated using an array that tracks active PIDs. When the number of background jobs reaches a predefined limit (e.g., 5), the script pauses new launches until a running job finishes, then decrements the queue length and starts the next task.

The algorithm can be expressed as:

Initialize an empty array pids=() and set MAX=5.

For each iteration, start the command in background and push its PID onto pids.

If ${#pids[@]} == MAX, enter a loop that checks wait -n (or polls kill -0) to detect a finished job, then removes its PID from the array.

After the main loop, call wait to ensure all remaining jobs complete.

Running this queue‑based script on the same 15‑task workload yields a total runtime of about 9 seconds (3 seconds × 3 batches), confirming that only five sleep processes run at any moment.

Example 3 – FIFO Pipe Token Mechanism

A named pipe (FIFO) can act as a token bucket. Create a FIFO file and preload it with Nproc tokens (e.g., write the number 5 lines). Each iteration reads one token before launching a background job; when the job finishes, it writes a token back, allowing another iteration to proceed.

Key steps:

Create FIFO: mkfifo /tmp/pipe.

Populate tokens: for i in $(seq 1 $Nproc); do echo > /tmp/pipe; done.

In the loop: read -u 3 (where file descriptor 3 is the FIFO) before starting the background command, and echo > /tmp/pipe after the command finishes.

This method also caps concurrent processes at Nproc while keeping the script simple and portable.

Summary and Recommendations

Parallelizing loop bodies dramatically reduces execution time, but uncontrolled concurrency can saturate I/O, network, or CPU resources. Use one of the following patterns based on your environment:

Simple background execution for quick, low‑impact tasks.

Array‑based queue simulation when you need a hard limit on simultaneous jobs.

FIFO token bucket for a lightweight, POSIX‑compatible throttling mechanism.

Avoid unbounded parallelism in production systems and prefer the queue or FIFO approaches as reusable templates for bulk operations.