Understanding Linux x86-64 System Call Implementation

In the Linux kernel, system calls are the primary mechanism for user‑space programs to request services from the kernel, and on the x86‑64 architecture they are invoked via the syscall instruction with the call number placed in rax and arguments in specific registers.

1. Overview of x86‑64 System Calls

System calls differ from ordinary function calls in that they trigger a mode switch from user to kernel, use a dedicated kernel stack, and rely on privileged instructions. The kernel assigns each call a unique number, which the CPU reads from rax to dispatch the appropriate handler.

2. Basic Knowledge

2.1 What is a System Call?

A system call provides a controlled interface for user programs to access hardware resources, file operations, process management, and other kernel services that cannot be performed directly from user space.

2.2 Hello‑World Example

.section.data
msg:
.ascii "Hello World!\n"
len = . - msg
.section.text
.globl	main
main:
    mov	$1, %rdi        # fd
    mov	$msg, %rsi      # buffer
    mov	$len, %rdx      # count
    mov	$1, %rax        # write syscall number (64‑bit)
    syscall
    mov	$0, %rdi        # status
    mov	$60, %rax       # exit syscall number
    syscall

Compiling and running this program demonstrates how arguments are placed in registers and how syscall transfers control to the kernel.

2.3 System Call Arguments

When using syscall , the first six arguments are passed in rdi, rsi, rdx, r10, r8, r9 respectively. The mapping is illustrated in the kernel source arch/x86/entry/entry_64.S and the table in the documentation.

/*
 * 64‑bit SYSCALL entry. Registers on entry:
 * rax  system call number
 * rcx  return address
 * r11  saved rflags
 * rdi  arg0
 * rsi  arg1
 * rdx  arg2
 * r10  arg3
 * r8   arg4
 * r9   arg5
 */

3. System Call Initialization

During kernel boot, start_kernel eventually calls syscall_init , which sets up the interrupt descriptor table entries (e.g., vector 0x80) and populates the system‑call table that maps numbers to handler functions.

3.1 start_kernel Flow

start_kernel → trap_init → cpu_init → syscall_init, establishing the linkage between interrupt vectors and the generic system_call entry point.

3.2 Role of Interrupt Vectors

Interrupt vectors translate a trap or syscall into a jump to the kernel’s entry routine, saving the user context and later restoring it after the handler finishes.

3.3 Wrapper Routines

Libc provides thin wrappers that place arguments in the correct registers and invoke syscall , sparing application developers from writing assembly.

4. System Call Table and Numbers

64‑bit system‑call numbers are defined in arch/x86/syscalls/syscall_64.tbl . Example entries: write is 1, exit is 60.

0   common  read            sys_read
1   common  write           sys_write
2   common  open            sys_open
3   common  close           sys_close
...
59  64      execve          sys_execve
60  common  exit            sys_exit

The kernel builds sys_call_table (in arch/x86/kernel/syscall_64.c ) as an array of function pointers indexed by these numbers.

5. System Call Handler

Entry points such as entry_SYSCALL_64 (for syscall ) and entry_INT80_32 (for int $0x80 ) save the CPU state, fetch the call number from rax , validate arguments, look up the handler in sys_call_table , invoke it, place the return value back in rax , and finally restore the saved registers.

6. Case Study: getpid

C Implementation

#include <stdio.h>
#include <unistd.h>

int main() {
    pid_t pid = getpid();
    printf("Current PID: %d\n", pid);
    return 0;
}

Assembly Implementation (x86‑64)

section.text
global _start

_start:
    mov rax, 39      ; sys_getpid
    syscall
    mov rdi, rax     ; PID -> rdi for write
    mov rax, 1       ; sys_write
    mov rsi, msg
    mov rdx, msg_len
    syscall
    mov rax, 60      ; sys_exit
    xor rdi, rdi
    syscall

section.data
msg db 'Current PID: ', 0
msg_len equ $ - msg

This demonstrates both high‑level and low‑level ways to invoke a system call on x86‑64.