6 Powerful Linux Commands to Find and Remove Duplicate Files

Both Windows and Linux computers accumulate duplicate files over time, which waste disk space and slow down the system. The following six methods show how to quickly identify and remove those duplicates.

1. Use diff command

The diff command compares two files and displays differences with < and > symbols; if there is no output, the files are identical.

<code>$ diff index.html backup.html
2438a2439,2441
> <pre>
> That's all there is to report.
>

When the files are the same:

$ diff home.html index.html

Note that diff can only compare two files at a time, making it inefficient for large sets.

2. Use checksum ( cksum )

cksum

computes a numeric checksum for each file; identical files share the same checksum.

$ cksum *.html
2819078353 228029 backup.html
4073570409 227985 home.html
4073570409 227985 index.html

3. Use find with cksum

Although find lacks a dedicated duplicate‑file option, it can locate files by name or type and pipe them to cksum.

$ find . -name "*.html" -exec cksum {} \;
4073570409 227985 ./home.html
2819078353 228029 ./backup.html
4073570409 227985 ./index.html

4. Use fslint

fslint

is a dedicated duplicate‑file finder. It requires installation and adding its directory to PATH.

$ fslint .
-----------------------------------file name lint
... 
----------------------------------DUPlicate files    <==
home.html
index.html

Tip: add fslint to the search path:

$ export PATH=$PATH:/usr/share/fslint/fslint

5. Use rdfind

rdfind

searches for redundant (identical) files and can automatically delete newer copies with the -deleteduplicates true option.

$ rdfind ~
Now scanning "/home/alvin", found 12 files.
Removed 1 files due to nonunique device and inode.
Total size 699498 bytes (683 KiB)

Dry‑run mode (no actual deletion):

$ rdfind -dryrun true ~
(DRYRUN MODE) Now scanning "/home/alvin", found 12 files.
...

Automatic deletion:

$ rdfind -deleteduplicates true .
Deleted 1 files.

6. Use fdupes

fdupes

lists duplicate files and offers many useful options; -r makes the search recursive.

$ fdupes ~
/home/alvin/UPGRADE
/home/alvin/mytwin

/home/alvin/lp.txt
/home/alvin/lp.man

/home/alvin/penguin.png
/home/alvin/penguin0.png
/home/alvin/hideme.png

Common options are summarized in the original article’s table (image omitted).

Conclusion

Linux provides a variety of tools— diff, cksum, find, fslint, rdfind, and fdupes —that enable you to locate and safely delete duplicate files, quickly freeing up disk space.