Debugging "directory not empty" errors from os.RemoveAll on a custom FUSE filesystem

Background : A Go program compiled for a custom distributed filesystem mounted via FUSE failed when calling os.RemoveAll(dir) on a non‑empty directory, reporting "directory not empty".

Investigation Process : The team examined the FUSE client implementation, noting that FUSE operates as a user‑space filesystem framework. They reproduced the issue by creating a directory with 300 files and running the removal script, which consistently triggered the error.

Root Cause : The kernel, during a readdir operation, received an interrupt signal and resent the request from the previous offset to /dev/fuse . The client incorrectly discarded the partially returned data, causing subsequent reads to fail because the expected offset data was missing. This manifested as incomplete unlink operations and the "directory not empty" error.

Detailed Trace : The kernel invokes sys_getdents64 , which leads to iterate_dir in VFS, then to fuse_readdir . The response is copied to user space via dir_emit . At a certain point, signal_pending(current) returned -4 , interrupting processing and forcing a retry from the original offset, exposing the bug.

Fix : After confirming the signal handling was normal, the team corrected the distributed filesystem logic to properly handle interrupted readdir calls and ensure all files are unlinked before removing the directory. The issue was resolved.

Kernel Module Build Commands :

make CONFIG_FUSE_FS=m M=fs/fuse
yes | cp fs/fuse/fuse.ko /usr/lib/modules/4.18.0_80.7.1.el8_0_bch_v1.0/kernel/fs/fuse/
umount /mnt
umount /sys/fs/fuse/connections/
rmmod fuse
insmod /usr/lib/modules/4.18.0_80.7.1.el8_0_bch_v1.0/kernel/fs/fuse/fuse.ko

Ftrace Debugging Commands :

cd /sys/kernel/debug/tracing
echo 0 > tracing_on
echo > trace
echo syscalls:* > set_event
echo raw_syscalls:* >> set_event
echo ${pid} >> set_event_pid
echo 1 > tracing_on
# run the program
cat trace

The experience shared aims to help others encountering similar FUSE‑related filesystem issues.