Why Go’s New Directory Traversal API Is Up to 5× Faster and How It Works

Go 1.23 is about to be released, but three and a half years ago Go 1.16 added a performance‑focused change to directory traversal.

Directory Traversal Optimization

Iterating over a directory is a common operation, especially for directories containing many files, where traversal speed directly impacts overall program performance.

Go 1.16 introduced new interfaces:

os.ReadDir
(*os.File).ReadDir
filepath.WalkDir

These interfaces use fs.DirEntry instead of os.FileInfo. fs.DirEntry provides methods similar to os.FileInfo and an Info() method to obtain the full os.FileInfo when needed.

type DirEntry interface {
    Name() string
    IsDir() bool
    Type() FileMode
    Info() (FileInfo, error)
}

A benchmark comparing the old Readdir (which calls os.Lstat for each entry) with the new ReadDir shows a 480% speedup on a test directory with 5,000 files on a Btrfs filesystem.

How the Optimization Works

The old implementation reads directory entries, then calls os.Lstat for each entry to obtain full metadata, which incurs many system calls and extra memory usage.

The new implementation changes the second argument of the internal f.readdir call to readdirDirEntry, allowing the kernel to return file name and type directly from the getdents system call (Linux) without extra lstat calls.

func (f *File) readdir(n int, mode readdirMode) (names []string, dirents []DirEntry, infos []FileInfo, err error) {
    // ... read directory entries via getdents ...
    if mode == readdirDirEntry {
        // construct DirEntry from the data already in the dirent
        de, err := newUnixDirent(f.name, name, direntType(rec))
        if err != nil {
            return nil, nil, nil, err
        }
        dirents = append(dirents, de)
    }
    // ...
}

newUnixDirent

builds a unixDirent struct containing the parent path, name, type, and optionally cached FileInfo. If the filesystem stores the file type in the directory entry, no extra lstat is needed; otherwise it falls back to lstat.

type unixDirent struct {
    parent string
    name   string
    typ    FileMode
    info   FileInfo
}

func newUnixDirent(parent, name string, typ FileMode) (DirEntry, error) {
    ude := &unixDirent{parent: parent, name: name, typ: typ}
    if typ != ^FileMode(0) && !testingForceReadDirLstat {
        return ude, nil
    }
    info, err := lstat(parent + "/" + name)
    if err != nil {
        return nil, err
    }
    ude.typ = info.Mode().Type()
    ude.info = info
    return ude, nil
}

The Info() method lazily calls lstat only when detailed metadata is required:

func (d *unixDirent) Info() (FileInfo, error) {
    if d.info != nil {
        return d.info, nil
    }
    return lstat(d.parent + "/" + d.name)
}

Thus, for simple traversals that only need names or types, the operation reduces to two system calls regardless of directory size, explaining the near‑five‑fold speedup.

Filesystem Support

The optimization relies on the filesystem storing the file type in the directory entry. It is supported by many common Linux filesystems:

btrfs, ext2, ext4 – fully supported.

OpenZFS – supported.

xfs – supported when created with mkfs.xfs -f -n ftype=1.

F2FS, EROFS – supported (checked in kernel source).

fat32, exfat – support limited to directory vs regular file.

ntfs – supports type detection but often falls back to an extra lstat, reducing the benefit.

Therefore, most mainstream filesystems on Linux and macOS can take advantage of the optimization.

Using the Optimization in Other Languages

Because the improvement is at the system‑call level, any language that uses readdir (or its equivalents) can benefit.

C : Call readdir directly; the kernel returns d_type when supported.

C++ : The standard library std::filesystem uses readdir internally and checks d_type similarly.

Python : os.scandir wraps readdir and provides .is_dir(), .is_file() without extra stat calls.

Conclusion

Go’s directory‑traversal overhaul demonstrates that low‑level system optimizations can yield massive performance gains without changing the public API. By leveraging filesystem‑provided metadata and reducing system‑call overhead, Go 1.16 achieves up to a five‑fold speed increase on supported filesystems, and the same principle applies to C, C++, and Python.