How to Parallelize Shell Loops on Linux Without External Tools

Why Serial Loops Are Slow

When managing dozens or hundreds of remote Linux hosts, administrators often need to run the same command on each host. A traditional for or while loop runs each iteration sequentially, so the total runtime equals the number of iterations multiplied by the per‑iteration cost, which can be unacceptable for long‑running tasks.

Running Loops in the Background (Example 1)

The simplest way to gain concurrency is to launch each iteration as a background job and wait for all jobs to finish at the end.

#!/bin/bash
Njob=15
for ((i=0; i<$Njob; i++)); do
  echo "progress $i is sleeping for 3 seconds zzz…"
  sleep 3 &
done
wait
echo -e "time-consuming: $SECONDS seconds"

Running this script reduces the total execution time from 45 seconds (15 × 3) to roughly 3 seconds, as all sleep commands run in parallel. However, this approach provides no limit on the number of concurrent processes, which can exhaust system resources in production.

Controlling Concurrency with a Simulated Queue (Example 2)

This method keeps track of child‑process IDs in an array‑based queue, allowing a maximum number of parallel jobs ( Nproc) to run at any time.

#!/bin/bash
Njob=15   # total tasks
Nproc=5   # max concurrent jobs

function PushQue { Que="$Que $1"; Nrun=$(($Nrun+1)); }
function GenQue { OldQue=$Que; Que=""; Nrun=0; for PID in $OldQue; do [[ -d /proc/$PID ]] && PushQue $PID; done; }
function ChkQue { OldQue=$Que; for PID in $OldQue; do [[ ! -d /proc/$PID ]] && GenQue && break; done; }

for ((i=1; i<=$Njob; i++)); do
  echo "progress $i is sleeping for 3 seconds zzz…"
  sleep 3 &
  PID=$!
  PushQue $PID
  while [[ $Nrun -ge $Nproc ]]; do
    ChkQue
    sleep 0.1
done
done
wait
echo -e "time-consuming: $SECONDS seconds"

The script launches up to five sleep processes simultaneously; when a process finishes, the queue is updated and a new task starts. This prevents resource exhaustion and avoids the “slow‑process” bottleneck.

Using a FIFO Token Mechanism (Example 3)

A named pipe (FIFO) can act as a token bucket. The pipe is pre‑filled with a number of tokens equal to the allowed concurrency. Each iteration reads a token before starting and writes one back when finished, automatically throttling the number of active jobs.

#!/bin/bash
Njob=15
Nproc=5
mkfifo ./fifo.$$ && exec 777<>./fifo.$$ && rm -f ./fifo.$$   # open FD 777 on FIFO
for ((i=0; i<$Nproc; i++)); do echo "init $i" >&777; done   # fill tokens

for ((i=0; i<$Njob; i++)); do
  {
    read -u 777   # acquire token
    echo "progress $i is sleeping for 3 seconds zzz…"
    sleep 3
    echo "real time add $((i+$Nproc))" >&777   # release token
  } &
done
wait
echo -e "time-consuming: $SECONDS seconds"

This approach guarantees that at most five processes run concurrently, and because the token is returned only after the child finishes, the overall runtime matches the longest individual task rather than being inflated by a “water‑bucket” effect.

Key Takeaways

Background execution ( &) provides raw speed but lacks any control over process count.

Maintaining a PID queue lets you cap concurrency while still using pure Bash.

A FIFO‑based token bucket offers a clean, race‑free way to enforce a maximum number of parallel jobs.

When scaling to many hosts, always consider system limits (open files, I/O, network bandwidth) and avoid launching more processes than the environment can safely handle.

Additional Notes

Even with controlled concurrency, process scheduling may cause out‑of‑order completion; for pure data‑collection tasks this is usually acceptable. For more deterministic ordering, consider adding a short random delay or using a post‑processing step such as sort or spreadsheet lookup functions.