How to Invoke Linux System Calls: glibc APIs, Direct syscall, and Inline Assembly

What is a system call?

A system call is an interface provided by the operating system that allows a user‑mode process to interact with hardware devices such as the CPU, disk, or printer. When a process invokes a system call, the CPU switches to kernel mode via a software interrupt and executes the corresponding kernel function.

Method 1: Using glibc library functions

glibc is the GNU implementation of the standard C library on Linux. It wraps kernel system calls in convenient API functions. Typically, each system call has at least one glibc wrapper (e.g., the kernel call sys_open is wrapped by the open function). Some glibc functions invoke multiple system calls (e.g., printf may call sys_write, sys_open, etc.), while several wrappers can map to the same system call (e.g., malloc, calloc, and free all use sys_brk).

Example: using the glibc chmod function to change /etc/passwd to mode 444. When run as an ordinary user, the call fails: chmod failed, errno = 1 The return value -1 indicates an error; the error code 1 is defined in /usr/include/asm‑generic/errno‑base.h as: #define EPERM 1 /* Operation not permitted */ Because a normal user lacks permission to modify /etc/passwd, the operation correctly fails.

Method 2: Directly invoking syscall

The syscall function, declared in unistd.h, lets a program invoke any kernel system call by number, bypassing glibc wrappers. Its prototype is:

long int syscall(long int sysno, ...)

sysno

– the unique system‑call number defined in sys/syscall.h. ... – the variable arguments representing the system‑call parameters (0‑5 arguments depending on the call).

Return value – the kernel’s return value; on error it returns -1 and sets errno.

Using syscall to perform the same chmod operation yields identical output when compiled and run as a normal user:

Method 3: Using the int 0x80 instruction (inline assembly)

If the full system‑call process is known, a program can trigger a kernel transition directly with the int 0x80 software interrupt (or sysenter on newer CPUs). The system‑call number is placed in eax, and up to five arguments are passed in ebx, ecx, edx, esi, and edi. After the kernel returns, the result is in eax; values between -1 and -132 represent error codes that must be stored in errno.

Below is an inline‑assembly snippet that changes the mode of /etc/passwd using the chmod system call (system‑call number 15 on 32‑bit Linux):

When compiled and executed as an ordinary user on a 32‑bit Linux system, this program produces the same error output as the previous two methods.

All three approaches demonstrate how to perform a system call in Linux, each with its own trade‑offs: glibc wrappers offer portability and ease of use; syscall provides direct access when a wrapper is missing; inline assembly gives maximum control but requires detailed knowledge of calling conventions.