Speed Up File Path Verification with Bash: Split & Parallel Execution

Background

In a production environment a MapReduce job generated a file list containing over 3 million lines, each line including a file path that needs to be checked for existence on servers outside the Hadoop cluster. A simple Bash script was initially considered but proved too slow.

Method 1 – Simple Linear Scan (Inefficient)

The first script reads the original file line by line, extracts the fifth field (the directory) with awk, tests the path with -e, and appends the line to exist.txt or noexist.txt accordingly.

#!/bin/bash
count=0
cat oriTest.txt | while read data; do
  count=$(( $count + 1 ))
  echo $count
  dir=$(echo "$data" | awk -F "\t" '{print $5}')
  if [ -e $dir ]; then
    echo "$data" >> exist.txt
  else
    echo "$data" >> noexist.txt
  fi
done

Processing 5,000 lines took nearly 4–5 minutes on an 8‑core machine, which was unacceptable.

Method 2 – Split File and Parallel Processing

The improved approach divides the large file into smaller pieces and processes each piece in the background.

Step 1: Split the Large File

split -l 10000 oriTest.txt

This creates files named xaa, xab, … each containing 10,000 lines.

Step 2: Store Chunk Names in an Array

declare -a files
files=($(ls x*))

Step 3: Define a Function to Process One Chunk

readdata(){
  cat $1 | while read data; do
    dir=$(echo "$data" | awk -F "\t" '{print $5}')
    if [ -e $dir ]; then
      echo "$data" >> "exist_$1.txt"
    else
      echo "$data" >> "noexist_$1.txt"
    fi
  done
}

Step 4: Launch Parallel Workers

for i in ${files[@]}; do
  echo $i
  readdata $i &
 done

Each readdata call runs in the background, allowing multiple chunks to be processed simultaneously and reducing total execution time dramatically.

Conclusion

By splitting the input file and executing the verification function in parallel, the script scales with the number of CPU cores and completes the existence check far faster than the single‑threaded version, making it suitable for large‑scale file‑path validation tasks.