Deep Dive into Linux VMA: Core Mechanisms of Virtual Memory Areas

1. What is Linux VMA?

In Linux virtual memory management, a VMA (Virtual Memory Area) is the fundamental unit the kernel uses to manage a process’s virtual address space. It partitions the address space into ordered regions such as code, data, stack, heap, shared libraries, and mmap mappings.

1.1 VMA Overview

A VMA is described by the struct vm_area_struct, which contains the start address ( vm_start) and end address ( vm_end) that define the region’s boundaries, as well as permission flags ( vm_flags) indicating read, write, and execute rights.

1.2 Why VMA is Needed

VMA simplifies address‑space management: without it, the kernel would have to handle a chaotic mix of code, data, and stacks. By grouping contiguous pages with the same attributes, the kernel can allocate, free, and protect memory efficiently, and it enables demand paging by tracking which pages are resident.

1.3 Relationship with Process Address Space

Each process has a struct mm_struct (memory descriptor) that holds pointers to all its VMAs. VMAs are linked in a doubly‑linked list for sequential traversal and inserted into a red‑black tree for O(log N) address lookup.

2. Core Data Structures

2.1 struct vm_area_struct

The VMA structure acts as an “identity card” for a memory region, recording its range, permissions, associated file (if any), and operation callbacks.

struct vm_area_struct {
    unsigned long vm_start;    /* start virtual address */
    unsigned long vm_end;      /* end virtual address (exclusive) */
    pgprot_t vm_page_prot;     /* page protection */
    unsigned long vm_flags;    /* VMA flags */
    struct file *vm_file;      /* mapped file */
    const struct vm_operations_struct *vm_ops; /* operation callbacks */
    struct vm_area_struct *vm_next;  /* next in list */
    struct rb_node vm_rb;               /* node in red‑black tree */
};

2.2 struct mm_struct and Its Link to VMA

The mm_struct is the “big manager” of a process’s address space. Its most important members are mmap (head of the VMA linked list) and mm_rb (root of the VMA red‑black tree).

struct mm_struct {
    struct vm_area_struct *mmap;   /* VMA list head */
    struct rb_root mm_rb;          /* VMA red‑black tree root */
    unsigned long start_code, end_code;   /* code segment */
    unsigned long start_data, end_data;   /* data segment */
    unsigned long start_brk, brk;         /* heap */
    unsigned long start_stack;             /* stack start */
};

2.3 Cooperation Between Linked List and Red‑Black Tree

The linked list is used for ordered traversal (e.g., printing all VMAs), while the red‑black tree provides fast lookup of the VMA that contains a given virtual address, reducing search time from O(N) to O(log N).

3. Core Mechanisms

3.1 Dynamic Allocation (mmap Example)

void *ptr = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

The kernel searches the VMA list and tree for a suitable free gap, creates a new vm_area_struct with the requested range, sets vm_flags to readable and writable, and inserts the VMA into both the list and the tree.

3.2 Access Control

When a process accesses memory, the kernel locates the responsible VMA via the red‑black tree and checks vm_flags. If the required permission (read, write, execute) is not set, the kernel raises SIGSEGV.

3.3 Page‑Fault Handling

If the accessed page is not resident, the kernel’s page‑fault handler calls find_vma to locate the VMA, verifies permissions, allocates a physical page (from per‑CPU caches or the buddy system), loads data from the mapped file if needed, updates the page table, and resumes the faulting instruction.

3.4 Copy‑on‑Write (COW)

During fork, the child receives a copy of the parent’s page tables, and writable private pages are marked read‑only in both processes. A write triggers a COW page‑fault, causing the kernel to allocate a new physical page, copy the contents, and update the child’s page table to writable.

4. Application Scenarios and Case Studies

4.1 Scenarios

Dynamic library loading : The dynamic linker uses mmap to map .so files, creating VMAs for code (read‑execute) and data (read‑write).

Memory allocation : malloc uses brk for small allocations (extending the heap VMA) and mmap for large allocations, each resulting in a new VMA.

4.2 Practical Code Example

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <string.h>

#define MAP_SIZE 4096

int main() {
    int fd;
    void *map_start;
    struct stat file_stat;
    char *hello = "Hello, VMA!";
    fd = open("test.txt", O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
    if (fd == -1) { perror("open"); return EXIT_FAILURE; }
    write(fd, "Initial content", strlen("Initial content"));
    if (fstat(fd, &file_stat) == -1) { perror("fstat"); close(fd); return EXIT_FAILURE; }
    map_start = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (map_start == MAP_FAILED) { perror("mmap"); close(fd); return EXIT_FAILURE; }
    printf("Content in mapped memory: %s
", (char *)map_start);
    strcpy((char *)map_start, hello);
    printf("Modified content in mapped memory: %s
", (char *)map_start);
    if (munmap(map_start, MAP_SIZE) == -1) { perror("munmap"); close(fd); return EXIT_FAILURE; }
    close(fd);
    return EXIT_SUCCESS;
}

This program opens a file, maps it into memory with mmap, reads and modifies the mapped region, then unmaps and closes the file.

4.3 Usage Considerations

Ensure vm_flags match actual accesses; mismatched permissions cause SIGSEGV.

Plan VMA layout to reduce fragmentation; excessive small VMAs increase management overhead.

Always pair mmap with munmap to avoid leaking virtual address space.

Limit page‑fault frequency by pre‑loading hot data or using huge pages.

When manipulating VMAs in kernel code, hold the appropriate memory semaphore (e.g., mmap_sem) and use kernel helpers like insert_vm_struct and remove_vm_struct.

For debugging VMA issues, inspect /proc/[pid]/maps and combine tools such as perf or valgrind to locate permission mismatches, illegal accesses, or memory leaks.