How Linux’s VFS Handles open() – Inside the Kernel’s File System Layers

Introduction

The virtual file system (VFS) in Linux abstracts concrete file‑system implementations, providing a uniform set of system calls that work on any mounted file system. This article uses the open() system call as a concrete entry point to explore VFS internals, based on the Linux 3.4.2 source tree.

Basic Knowledge

Linux supports three families of file systems: disk‑based, memory‑based, and network file systems. VFS sits between user space and these families, offering a single I/O interface regardless of the underlying format.

VFS Data Structures

Superblock

struct super_block {
    struct list_head s_list;          // list of superblocks
    struct file_system_type *s_type; // file‑system type
    struct super_operations *s_op;   // superblock methods
    struct list_head s_instances;   // instances of this type
    ...
};

struct super_operations {
    struct inode *(*alloc_inode)(struct super_block *sb);
    void (*read_inode)(struct inode *);
    ...
};

Inode

struct inode {
    struct inode_operations *i_op;   // inode operation table
    struct file_operations *i_fop;   // file operations for this inode
    struct super_block *i_sb;        // associated superblock
    ...
};

struct inode_operations {
    int (*create)(struct inode *, struct dentry *, int, struct nameidata *);
    struct dentry *(*lookup)(struct inode *, struct dentry *, struct nameidata *);
    ...
};

Dentry

struct dentry {
    struct inode *d_inode;          // linked inode
    struct dentry *d_parent;       // parent directory
    struct qstr d_name;            // name of the entry
    struct list_head d_subdirs;    // child directories
    struct dentry_operations *d_op; // dentry operation table
    struct super_block *d_sb;      // superblock of the filesystem
    ...
};

struct dentry_operations {
    int (*d_revalidate)(struct dentry *, struct nameidata *);
    int (*d_hash)(struct dentry *, struct qstr *);
    ...
};

File

struct file {
    struct list_head f_list;        // file object list
    struct dentry *f_dentry;      // associated dentry
    struct vfsmount *f_vfsmnt;     // mounted filesystem
    struct file_operations *f_op;   // file operation table
    ...
};

struct file_operations {
    ssize_t (*read)(struct file *, char __user *, size_t, loff_t *);
    ssize_t (*write)(struct file *, const char __user *, size_t, loff_t *);
    int (*open)(struct inode *, struct file *);
    int (*readdir)(struct file *, void *, filldir_t);
    ...
};

open() System Call Flow

Application Layer

User programs invoke open() with a pathname, flags, and mode. The C library forwards these arguments to the kernel.

int open(const char *pathname, int oflag, mode_t mode);

Kernel Layer

The system call is handled by

SYSCALL_DEFINE3(open, const char __user *filename, int flags, int mode)

in fs/open.c. It adjusts flags for large‑file support and then calls do_sys_open().

SYSCALL_DEFINE3(open, const char __user *filename, int flags, int mode)
{
    long ret;
    if (force_o_largefile())
        flags |= O_LARGEFILE;
    ret = do_sys_open(AT_FDCWD, filename, flags, mode);
    asmlinkage_protect(3, ret, filename, flags, mode);
    return ret;
}

do_sys_open()

copies the user‑space filename into kernel memory, obtains an unused file descriptor, and creates a file object via do_filp_open(). The new file object is linked into the process’s file‑descriptor table.

long do_sys_open(int dfd, const char __user *filename, int flags, int mode)
{
    char *tmp = getname(filename);
    int fd = PTR_ERR(tmp);
    if (!IS_ERR(tmp)) {
        fd = get_unused_fd();
        if (fd >= 0) {
            struct file *f = do_filp_open(dfd, tmp, flags, mode, 0);
            if (IS_ERR(f)) {
                put_unused_fd(fd);
                fd = PTR_ERR(f);
            } else {
                fsnotify_open(f);
                fd_install(fd, f);
            }
        }
        putname(tmp);
    }
    return fd;
}

The created file object contains a pointer to a dentry, which in turn points to an inode. The inode links back to its super_block, completing the chain from user‑level file descriptor to the actual storage representation.

Conclusion

The VFS layer lets Linux present a consistent open() interface for all file‑system types. By mapping a file descriptor to a file object, then to a dentry and finally to an inode, the kernel can locate and manipulate the underlying file data on disk, in memory, or over the network.