How to Build a Python Script that Detects and Removes Duplicate Files

Introduction

Hello everyone, it's time for the Python office automation series. This article presents a system‑level automation case: given a folder, use Python to check for duplicate files and delete them.

Key Modules

os

– comprehensive usage glob – comprehensive usage filecmp – compare two files

Step Analysis

The program traverses all files in the target folder, compares each pair, and deletes the latter when duplicates are found.

Traverse the folder, compare files pairwise, delete duplicates.

The crucial question is how to determine if two files are identical. The filecmp module provides filecmp.cmp(f1, f2, shallow=True), which returns True if files are considered equal (based on os.stat() when shallow is true) or False otherwise.

# Assume x and y are identical files
print(filecmp.cmp(x, y))
# True

Python Implementation

Import libraries and set the target directory:

import os
import glob
import filecmp

dir_path = r'C:\xxxx'

Collect absolute paths of all files using glob with recursive=True:

for file in glob.glob(dir_path + '/**/*', recursive=True):
    pass

Build a list of file paths:

file_lst = []
for i in glob.glob(dir_path + '/**/*', recursive=True):
    if os.path.isfile(i):
        file_lst.append(i)

Compare each pair with filecmp.cmp and delete duplicates, checking existence to avoid errors:

for x in file_lst:
    for y in file_lst:
        if x != y and os.path.exists(x) and os.path.exists(y):
            if filecmp.cmp(x, y):
                os.remove(y)

The complete script combines the above steps.

Conclusion

This simple duplicate‑file remover demonstrates the power of Python for office automation and can be combined with other file‑organizing scripts.