How to Quickly Find Duplicate Files on Linux Using Find and dupeGuru

Method 1: Using the Find Command

This section extends the powerful find utility by combining it with other basic Linux commands such as xargs to create an unlimited range of command‑line functions, for example quickly listing duplicate files in a directory tree.

find -not -empty -type f -printf "%s
" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

find -not -empty -type f -printf "%s\n" – lists all non‑empty regular files and prints their sizes.

sort -rn – sorts the sizes in reverse numeric order.

uniq -d – keeps only the duplicated size entries.

uniq -w32 --all-repeated=separate – compares the first 32 bytes of the MD5 hash to filter duplicate files.

Method 2: Using the dupeGuru Tool

dupeGuru is a cross‑platform application (Linux, Windows, macOS) that can locate duplicate files by size, MD5, or filename. Ubuntu users can install it via a PPA:

sudo add-apt-repository ppa:hsoft/ppa
sudo apt-get update
sudo apt-get install dupeguru*

Method 3: Using Find Command Parsing

When you need to find duplicate text files (e.g., game scripts) for translation, a shell script can automate the process.

Copy the following pipeline into the target directory:

find -not -empty -type f -printf "%s
" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate | cut -b 36-

The pipeline works as follows: find -not -empty -type f -printf "%s\n" Outputs the size of every non‑empty regular file. sort -rn Sorts sizes in descending order. uniq -d Keeps only duplicated size values. xargs -I{} -n1 find -type f -size {}c -print0 Finds files matching each duplicated size, using a null‑terminated list to handle spaces. xargs -0 md5sum Computes MD5 hashes for those files. uniq -w32 --all-repeated=separate Groups lines with identical first 32 characters (the MD5 hash). cut -b 36- Removes the MD5 part to keep only the file paths.

Store the final output in result.txt:

find -not -empty -type f -printf "%s
" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate | cut -b 36- > result.txt

On Windows, convert Linux line endings (\n) to Windows style (\r\n):

cat result.txt | cut -c 36- | tr -s '
'