How to Hook Linux System Calls for Real‑Time Monitoring and Security

1. Linux System Call Overview

System calls are the core channel through which user‑space programs interact with the kernel, enabling operations such as file I/O, process creation, and memory allocation.

1.1 Concept of System Calls

A system call is an interface provided by the kernel that user programs invoke when they need to access hardware resources, create processes, allocate memory, and perform other privileged operations.

1.2 Implementation Mechanism

On Linux, system calls are invoked via software interrupts (e.g., int 0x80 on x86 or sysenter on newer kernels). The CPU switches to kernel mode, looks up the call number in the syscall_table, and jumps to the corresponding kernel function.

2. Hook Technology Details

2.1 Ring 3 Hook Techniques

LD_PRELOAD dynamic linking intercepts functions by loading a custom shared object before any other libraries. The loader searches /etc/ld.so.cache, /etc/ld.so.preload, LD_LIBRARY_PATH, and the binary’s RPATH to locate libraries.

Update /etc/ld.so.cache with ldconfig after adding or removing shared libraries.

Specify libraries in /etc/ld.so.preload for system‑wide preloading.

Example hook.c that overrides strcmp:

#include <stdio.h>
#include <string.h>
#include <dlfcn.h>

typedef int(*STRCMP)(const char*, const char*);

int strcmp(const char *s1, const char *s2)
{
    static void *handle = NULL;
    static STRCMP old_strcmp = NULL;
    if (!handle) {
        handle = dlopen("libc.so.6", RTLD_LAZY);
        old_strcmp = (STRCMP)dlsym(handle, "strcmp");
    }
    printf("oops!!! hack function invoked. s1=<%s> s2=<%s>
", s1, s2);
    return old_strcmp(s1, s2);
}

Compile and use:

gcc -o test main.c
gcc -fPIC -shared -o hook.so hook.c -ldl
LD_PRELOAD=./hook.so ./test 123

2.2 Ring 0 Hook Techniques

Kernel‑space hooking modifies the kernel code directly, for example by patching the opcode of sys_read to jump to a custom function, or by using kprobes to insert breakpoints.

Inline hook example (simplified):

unsigned int patch_kernel_func(unsigned int handler, unsigned int old_func, unsigned int new_func)
{
    unsigned char *p = (unsigned char *)handler;
    unsigned char buf[4];
    unsigned int offset, orig;
    // Search for call opcode (0xe8) and replace offset
    // ...
    return orig;
}

kprobe registration example:

#include <linux/module.h>
#include <linux/kprobes.h>

static struct kprobe kp = {.symbol_name = "__x64_sys_open"};

static int pre_handler(struct kprobe *p, struct pt_regs *regs)
{ printk(KERN_INFO "MyHook: Before open syscall
"); return 0; }

static void post_handler(struct kprobe *p, struct pt_regs *regs, unsigned long flags)
{ printk(KERN_INFO "MyHook: After open syscall
"); }

static int __init my_init(void)
{ kp.pre_handler = pre_handler; kp.post_handler = post_handler; return register_kprobe(&kp); }

static void __exit my_exit(void) { unregister_kprobe(&kp); }

module_init(my_init);
module_exit(my_exit);
MODULE_LICENSE("GPL");

3. Common Hook Methods for Linux System Calls

3.1 Function‑Pointer Hook

Directly replace entries in syscall_table with pointers to custom functions, then optionally call the original function via a saved pointer.

3.2 GOT‑Based Hook

In user‑space, modify the Global Offset Table of a dynamically linked binary (often via LD_PRELOAD) so that calls to libc functions are redirected to custom implementations.

3.3 Kprobe Hook

Register a kprobe on a kernel function (e.g., __x64_sys_open) and provide pre‑ and post‑handlers to execute custom logic without altering kernel code.

4. Hook Challenges and Mitigations

4.1 Kernel Stability

Incorrect hook code can crash the kernel. Mitigation: extensive testing, backup modified memory regions, and graceful rollback on failure.

4.2 Compatibility

Kernel versions and architectures differ in syscall table layout. Mitigation: use version‑specific macros, runtime discovery of sys_call_table, and conditional compilation.

4.3 Security Risks

Malicious actors can abuse hooks to hide activity. Mitigation: enforce strict permission checks, audit hook installations, and employ detection tools that monitor unexpected modifications to kernel structures.