Why Go’s New Directory Traversal API Is Up to 5× Faster and How It Works

Go 1.23 is about to be released, but three and a half years ago Go 1.16 added a performance‑focused change to directory traversal.

Directory Traversal Optimization

Iterating over a directory is a common operation, especially for directories containing many files, where traversal speed directly impacts overall program performance.

Go 1.16 introduced new interfaces:

os.ReadDir
(*os.File).ReadDir
filepath.WalkDir

These interfaces use

fs.DirEntry

instead of

os.FileInfo

.

fs.DirEntry

provides methods similar to

os.FileInfo

and an

Info()

method to obtain the full

os.FileInfo

when needed.

<code>type DirEntry interface {
    Name() string
    IsDir() bool
    Type() FileMode
    Info() (FileInfo, error)
}
</code>

A benchmark comparing the old

Readdir

(which calls

os.Lstat

for each entry) with the new

ReadDir

shows a 480% speedup on a test directory with 5,000 files on a Btrfs filesystem.

How the Optimization Works

The old implementation reads directory entries, then calls

os.Lstat

for each entry to obtain full metadata, which incurs many system calls and extra memory usage.

The new implementation changes the second argument of the internal

f.readdir

call to

readdirDirEntry

, allowing the kernel to return file name and type directly from the

getdents

system call (Linux) without extra

lstat

calls.

<code>func (f *File) readdir(n int, mode readdirMode) (names []string, dirents []DirEntry, infos []FileInfo, err error) {
    // ... read directory entries via getdents ...
    if mode == readdirDirEntry {
        // construct DirEntry from the data already in the dirent
        de, err := newUnixDirent(f.name, name, direntType(rec))
        if err != nil {
            return nil, nil, nil, err
        }
        dirents = append(dirents, de)
    }
    // ...
}
</code>

newUnixDirent

builds a

unixDirent

struct containing the parent path, name, type, and optionally cached

FileInfo

. If the filesystem stores the file type in the directory entry, no extra

lstat

is needed; otherwise it falls back to

lstat

.

<code>type unixDirent struct {
    parent string
    name   string
    typ    FileMode
    info   FileInfo
}

func newUnixDirent(parent, name string, typ FileMode) (DirEntry, error) {
    ude := &unixDirent{parent: parent, name: name, typ: typ}
    if typ != ^FileMode(0) && !testingForceReadDirLstat {
        return ude, nil
    }
    info, err := lstat(parent + "/" + name)
    if err != nil {
        return nil, err
    }
    ude.typ = info.Mode().Type()
    ude.info = info
    return ude, nil
}
</code>

The

Info()

method lazily calls

lstat

only when detailed metadata is required:

<code>func (d *unixDirent) Info() (FileInfo, error) {
    if d.info != nil {
        return d.info, nil
    }
    return lstat(d.parent + "/" + d.name)
}
</code>

Thus, for simple traversals that only need names or types, the operation reduces to two system calls regardless of directory size, explaining the near‑five‑fold speedup.

Filesystem Support

The optimization relies on the filesystem storing the file type in the directory entry. It is supported by many common Linux filesystems:

btrfs, ext2, ext4 – fully supported.

OpenZFS – supported.

xfs – supported when created with

mkfs.xfs -f -n ftype=1

.

F2FS, EROFS – supported (checked in kernel source).

fat32, exfat – support limited to directory vs regular file.

ntfs – supports type detection but often falls back to an extra

lstat

, reducing the benefit.

Therefore, most mainstream filesystems on Linux and macOS can take advantage of the optimization.

Using the Optimization in Other Languages

Because the improvement is at the system‑call level, any language that uses

readdir

(or its equivalents) can benefit.

C : Call

readdir

directly; the kernel returns

d_type

when supported.

C++ : The standard library

std::filesystem

uses

readdir

internally and checks

d_type

similarly.

Python :

os.scandir

wraps

readdir

and provides

.is_dir()

,

.is_file()

without extra

stat

calls.

Conclusion

Go’s directory‑traversal overhaul demonstrates that low‑level system optimizations can yield massive performance gains without changing the public API. By leveraging filesystem‑provided metadata and reducing system‑call overhead, Go 1.16 achieves up to a five‑fold speed increase on supported filesystems, and the same principle applies to C, C++, and Python.