How GPU Kernel Virtualization Works: A Deep Dive into the cgpu Project

Kernel Interception

In Linux a device is accessed through a file descriptor, so applications interact with devices such as /dev/nvidia0 via system calls open, read, write, mmap and ioctl. The driver registers a file_operations structure that maps these calls to driver functions.

To observe which system calls are issued when using an NVIDIA device, the strace command can be used:

$ strace -e trace=mmap nvidia-smi
$ strace -e trace=ioctl nvidia-smi
$ strace nvidia-smi

cgpu Core Framework

The cgpu source consists of three files: cgpu-km.c, cgpu-procfs.c and os-interface.c. The entry point is cgpu-km.c, which provides the module’s init and exit functions.

cgpu‑km.c analysis

The driver registers a file_operations structure named cgpu_km_fops that implements only a few callbacks, indicating that these functions are sufficient for virtualization:

static struct file_operations cgpu_km_fops = {
    .owner          = THIS_MODULE,
    .poll           = cgpu_km_poll,
    .unlocked_ioctl = cgpu_km_unlocked_ioctl,
    .compat_ioctl   = cgpu_km_compat_ioctl,
    .mmap           = cgpu_km_mmap,
    .open           = cgpu_km_open,
    .release        = cgpu_km_close,
};

The actual implementations of these callbacks are defined in cgpu.o and can be inspected with objdump -t cgpu.o.

During module loading, cgpu_km_init performs the following steps:

static int __init cgpu_km_init(void)
{
    int ret = 0;
    int num = 0;
    num = cgpu_initialize();               // defined in cgpu.o
    ret = register_chrdev(0, "cgpu-km", &cgpu_km_fops); // register character device
    cgpu_major = ret;
    ret = cgpu_km_procfs_init(num);        // defined in cgpu-procfs.c
    if (ret) {
        pr_err("failed to init cgpu km procfs
");
        unregister_chrdev(cgpu_major, "cgpu-km");
    }
    return ret;
}

When the module is removed, cgpu_km_exit cleans up resources:

static void __exit cgpu_km_exit(void)
{
    cgpu_finalize();               // defined in cgpu.o
    cgpu_km_procfs_deinit();       // defined in cgpu-procfs.c
    unregister_chrdev(cgpu_major, "cgpu-km");
}

cgpu‑procfs.c analysis

cgpu uses procfs (instead of sysfs) to expose kernel‑mode parameters to user space.

os‑interface.c analysis

This file implements the cGPU scheduling algorithm with a kernel thread and a queue, enabling multi‑task switching on a single GPU.

Compilation

The project was successfully compiled on Ubuntu 22.04 with a newer kernel; a few deprecated system calls were updated. Modified source and build instructions are available at:

https://github.com/lengrongfu/study-demo/blob/main/gpu/cgpu/README.md

References

Detailed analysis article:

https://www.cnblogs.com/shaohef/p/13616687.html