Unlock Hidden Unix Commands: Find Missing Files with seq, grep, cut, and comm

When running many simulations for a master's thesis, each dataset generates files like dataset-directory/0001_data.csv and dataset-directory/0001_A.csv. Some runs fail, and the goal is to list the dataset numbers where algorithm A did not produce a result.

Solution Overview

The missing numbers can be obtained by subtracting the list of successful runs from the full range (1‑500). The seq command generates the complete sequence, while a pipeline of ls, grep, cut, and a small Python script extracts the numbers of successful A files.

Generate the full list

$ seq 500

Extract successful A files

$ ls dataset-directory | grep '\d\d\d\d_A\.csv' | sort | cut -c 1-4 | python3 - <<'PY'
import sys
for line in sys.stdin:
    print(int(line))
PY

This pipeline lists all files, filters those ending with _A.csv, sorts them, cuts the leading four digits, and converts them to integers.

Find missing numbers with comm

The comm utility compares two sorted inputs. Using process substitution, we compare the successful list with the full sequence, suppressing the first and third columns (numbers present in both inputs) to keep only the missing ones:

$ comm -1 -3 <(ls dataset-directory | grep '\d\d\d\d_A\.csv' | cut -c 1-4 | python3 parse.py) <(seq 500)

The output lists dataset numbers such as 4, 8, … that lack an A result.

Key Unix Tools Demonstrated

seq : generate numeric sequences.

ls + grep : list files matching a pattern.

sort : ensure inputs are ordered for comm.

cut : extract the numeric prefix.

python3 : convert zero‑padded strings to integers.

comm : compare two sorted streams and output unique lines.

These commands illustrate the Unix philosophy of building complex workflows by chaining simple, single‑purpose tools.