How Go 1.16’s Directory Traversal Optimization Boosts Performance by 480%

Go 1.16 added three new directory‑traversal functions— os.ReadDir, (*os.File).ReadDir and filepath.WalkDir —that return fs.DirEntry instead of the heavyweight os.FileInfo. The fs.DirEntry interface provides only the file name, type and an Info() method for on‑demand os.FileInfo retrieval.

Directory Traversal Optimization

Benchmarking a 5 000‑file directory on a Btrfs filesystem shows the new APIs are about 480 % faster than the classic Readdir + os.Lstat loop because the new implementation reads directory entries with a single getdents system call and avoids per‑file lstat calls unless additional metadata is required.

type DirEntry interface {
    Name() string
    IsDir() bool
    Type() FileMode
    Info() (FileInfo, error)
}

The old implementation called os.Lstat for every entry, which caused thousands of extra system calls and memory allocations. The new implementation passes a different mode to the internal f.readdir function ( readdirDirEntry), allowing it to construct unixDirent objects directly from the data returned by getdents without extra lstat calls.

type unixDirent struct {
    parent string
    name   string
    typ    FileMode
    info   FileInfo
}

func newUnixDirent(parent, name string, typ FileMode) (DirEntry, error) {
    ude := &unixDirent{parent: parent, name: name, typ: typ}
    if typ != ^FileMode(0) && !testingForceReadDirLstat {
        return ude, nil
    }
    info, err := lstat(parent + "/" + name)
    if err != nil {
        return nil, err
    }
    ude.typ = info.Mode().Type()
    ude.info = info
    return ude, nil
}

If the underlying filesystem stores the file type in the directory entry (most modern Linux filesystems do), the optimization is fully effective; otherwise the code falls back to lstat. Supported filesystems include btrfs, ext2/4, XFS (with the ftype=1 option), F2FS, EROFS, FAT family, and NTFS (with reduced benefit).

Cross‑Language Usage

Because the optimization lives in the kernel’s readdir implementation, any language that uses the POSIX readdir (C, C++, Python’s os.scandir, etc.) can benefit automatically, provided the filesystem supports the d_type field.

Conclusion

Go 1.16’s directory‑traversal improvement demonstrates how a small change—leveraging existing kernel metadata and reducing system‑call overhead—can yield a near‑five‑fold speedup for large directories, and the same principle applies to other languages and platforms.