Understanding Linux System Calls: Theory, Mechanisms, and Practical Examples

Part1 Linux System Calls: What Are They?

In the vast world of Linux, system calls act as a crucial bridge between user programs and the kernel, providing the only legitimate way for applications to request kernel services such as file operations, process management, and memory allocation.

System calls are exposed as C functions, simplifying access to kernel functionality while protecting hardware resources from direct user‑space access, ensuring stability and security.

1.1 Definition of System Calls

System calls are the primary interface for user programs to interact with the kernel, allowing access to core services like process, memory, and file‑system management.

1.2 Importance of System Calls

They form the foundation of OS services, handling resource management, process control, file operations, and device management.

In resource management, system calls let the kernel allocate memory, CPU time slices, and file descriptors, ensuring fair and efficient usage.

For process control, calls such as fork(), wait(), and waitpid() enable creation, synchronization, and termination of processes, facilitating concurrent programming.

File operations are unified through calls like open(), read(), write(), and close(), abstracting underlying file‑system details.

Device management uses calls such as ioctl() to configure and interact with hardware devices.

1.3 Why System Calls Are Needed

System calls run in kernel mode, providing a protected entry point for user‑space programs to request privileged operations while preserving system stability and security.

They offer a uniform hardware‑abstraction interface for user programs.

They enforce security checks and permission validation.

They maintain a clear separation between user and kernel space, enabling multitasking and virtual memory.

Part2 User Space and Kernel Space

On a 32‑bit Linux system, the lower 3 GB of virtual address space is user space, while the upper 1 GB is reserved for the kernel. User programs run with limited privileges and cannot directly access hardware resources; they must request services via system calls.

2.1 User Space

User space provides an isolated execution environment for each process, preventing crashes in one process from affecting others.

2.2 Kernel Space

Kernel space holds the core of the operating system, with full control over CPU, memory, and I/O devices, handling scheduling, memory management, file systems, and drivers.

2.3 Why Separate Spaces?

Separating privileged kernel instructions (Ring 0) from unprivileged user instructions (Ring 3) protects the system from accidental or malicious misuse of critical operations.

Part3 Execution Process of a System Call

3.1 Preparation

Before invoking a system call, the program sets the system‑call number and any required arguments in registers (e.g., EAX for the call number, EBX, ECX, EDX for parameters on x86).

3.2 Triggering the Call

On x86, the int 0x80 software interrupt or the newer syscall instruction transfers control to the kernel. Other architectures use their own mechanisms (e.g., svc on ARM).

3.3 Entering Kernel Mode

The CPU saves the user context, switches to kernel mode, and jumps to the system‑call handler via the interrupt vector table.

3.4 Parameter Checking

The kernel validates argument types, ranges, and pointer legitimacy to prevent crashes and security breaches.

3.5 Executing the Kernel Function

Using the call number, the kernel looks up the corresponding function in the system‑call table and executes it, handling up to six arguments passed in registers.

3.6 Returning to User Mode

After execution, the kernel places the return value in a register (e.g., EAX) and restores the saved user context, allowing the program to continue.

Part4 Three Ways to Invoke System Calls in Linux

4.1 Using glibc Wrapper Functions

glibc provides high‑level APIs (e.g., open(), chmod()) that internally invoke the appropriate system calls. Multiple wrappers may map to the same call, and a single wrapper may use several calls.

#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <stdio.h>

int main() {
    int rc = chmod("/etc/passwd", 0444);
    if (rc == -1)
        fprintf(stderr, "chmod failed, errno = %d
", errno);
    else
        printf("chmod success!
");
    return 0;
}

Running as a normal user yields chmod failed, errno = 1 (EPERM).

4.2 Directly Using syscall()

The syscall function lets you invoke any system call by number, useful when glibc lacks a wrapper.

#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <errno.h>

int main() {
    int rc = syscall(SYS_chmod, "/etc/passwd", 0444);
    if (rc == -1)
        fprintf(stderr, "chmod failed, errno = %d
", errno);
    else
        printf("chmod success!
");
    return 0;
}

4.3 Using Inline Assembly with int 0x80

When the full call flow is known, you can issue the interrupt directly.

#include <stdio.h>
#include <sys/types.h>
#include <sys/syscall.h>
#include <errno.h>

int main() {
    long rc;
    char *file_name = "/etc/passwd";
    unsigned short mode = 0444;
    asm(
        "int $0x80"
        : "=a" (rc)
        : "0" (SYS_chmod), "b" ((long)file_name), "c" ((long)mode)
    );
    if ((unsigned long)rc >= (unsigned long)-132) {
        errno = -rc;
        rc = -1;
    }
    if (rc == -1)
        fprintf(stderr, "chmod failed, errno = %d
", errno);
    else
        printf("chmod success!
");
    return 0;
}

Part5 Case Studies of System Calls

5.1 Implementation Details

Linux maintains a system‑call table where each entry’s index is the call number. The kernel dispatches calls based on the number placed in EAX. Arguments are passed via registers ( EBX, ECX, EDX, ESI, EDI, EBP) and the stack is switched from user to kernel space during the transition.

5.2 File‑Operation System Calls

Typical calls include open(), read(), write(), and close(). A complete example reads from source.txt and writes to destination.txt using these calls.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>

#define BUFFER_SIZE 1024

int main() {
    int src = open("source.txt", O_RDONLY);
    if (src == -1) { perror("open source"); return 1; }
    int dst = open("destination.txt", O_WRONLY|O_CREAT|O_TRUNC,
                   S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH);
    if (dst == -1) { perror("open dest"); close(src); return 1; }
    char buf[BUFFER_SIZE];
    ssize_t n;
    while ((n = read(src, buf, BUFFER_SIZE)) > 0) {
        if (write(dst, buf, n) != n) { perror("write"); close(src); close(dst); return 1; }
    }
    if (n == -1) perror("read");
    close(src); close(dst);
    return 0;
}

5.3 Process‑Management System Calls

Calls such as fork(), exec*, and wait() enable process creation, program replacement, and synchronization. The following program demonstrates creating a child process, executing ls -l, and waiting for termination.

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdlib.h>

int main() {
    pid_t pid = fork();
    if (pid == -1) { perror("fork"); exit(1); }
    else if (pid == 0) {
        printf("Child PID %d, parent PID %d
", getpid(), getppid());
        execl("/bin/ls", "ls", "-l", NULL);
        perror("execl"); exit(1);
    } else {
        printf("Parent PID %d, child PID %d
", getpid(), pid);
        int status; wait(&status);
        printf("Child exited with status %d
", WEXITSTATUS(status));
    }
    return 0;
}