How Linux Buffer Overflows Work and How to Defend Against Them

Understanding Buffer Overflow on Linux

After reviewing the layout of a Linux process address space and stack‑frame structure, the article presents a simple program example2.c that manipulates the return address of a function to change the program’s control flow. The function adds 10 to the saved return address, causing the main function to skip the instruction that sets x to 1, so the printed result remains 0.

Disassembly of main shows the call instruction at 0x804833f pushing 0x8048344 onto the stack, which the malicious line (*ret) += 10; changes to 0x804834e. When function returns, execution jumps to the modified address, bypassing the mov that would set x to 1.

Crafting Shellcode

The article then introduces shellcode – a small piece of code that spawns a shell – and walks through the creation of a minimal Linux Hello world program using direct system calls via int $0x80. The relevant system‑call numbers ( __NR_write = 4, __NR_exit = 1) are taken from /usr/include/asm/unistd.h.

#include <asm/unistd.h>
int errno;
_syscall3(int, write, int, fd, char *, data, int, len);
_syscall1(int, exit, int, status);
_start() {
    write(0, "Hello world!
", 13);
    exit(0);
}

Compiling and linking produces a tiny executable ( hello ) whose size can be compared with a traditional C printf version.

Next, the article shows how to build a shellcode that invokes execve("/bin/sh", NULL, NULL) using the same syscall mechanism. The initial assembly version ( shellcodeasm.c) stores the /bin/sh string in the code segment, sets up registers ( eax=11, ebx=<addr>, ecx=<addr>, edx=0), and triggers the interrupt.

#include <unistd.h>
int main() {
    char *name[2];
    name[0] = "/bin/sh";
    name[1] = NULL;
    execve(name[0], name, NULL);
    _exit(0);
}

Because the code segment is non‑writable, executing the original version causes a segmentation fault. The article moves the code to a writable data segment or stack, then refines the assembly to remove null bytes and shorten instructions. The final optimized shellcode is presented both as assembly ( shellcodeasm2.c) and as a byte string:

char shellcode[] = "\xeb\x1f\x5e\x89\x76\x08\x31\xc0\x88\x46\x07\x89\x46\x0c\xb0\x0b"
"\x89\xf3\x8d\x4e\x08\x8d\x56\x0c\xcd\x80\x31\xdb\x89\xd8\x40\xcd"
"\x80\xe8\xdc\xff\xff\xff/bin/sh";

Testing the shellcode with a small driver program ( testsc.c) demonstrates that the injected code successfully spawns a shell.

Full Exploit Example

The article revisits the original buffer‑overflow example, describing how an attacker can use command‑line arguments and environment variables (e.g., KIRIKA) to place the shellcode in a vulnerable program’s stack, overwrite the return address, and gain a root shell when the vulnerable program ( toto.c) is setuid‑root.

Mitigation with Libsafe

To defend against such attacks, the article introduces Libsafe , a library that replaces unsafe string functions (e.g., strcpy, strcat, gets) with safer alternatives via the LD_PRELOAD mechanism. By preloading libsafe.so, calls to vulnerable functions are intercepted and checked for buffer overflows.

Example vulnerable program ( testlibsafe.c) copies a 16‑byte buffer into an 8‑byte buffer, causing a segmentation fault. After installing Libsafe (RPM libsafe-2.0-2.i386.rpm) and setting LD_PRELOAD=/lib/libsafe.so.2, the library detects the overflow, prints diagnostic information (UID, PID, call stack), and logs the event.

#include <string.h>
void main() {
    char buf1[8];
    char buf2[16];
    int i;
    for (i = 0; i < 16; ++i)
        buf2[i] = 'A';
    strcpy(buf1, buf2);
}

Static linking bypasses Libsafe because the preload mechanism works only with dynamic libraries. The article demonstrates this limitation by compiling the same program with -static, which still crashes despite Libsafe being installed.

Advanced Considerations

Modern Bash versions drop root privileges after a setuid‑root program spawns a shell by calling seteuid(getuid()). To regain root, the shellcode must invoke setuid(0) before execve. Updated shellcode examples ( shellcodeasm3.c and exe_pro.c) illustrate this technique.

Overall, the article provides a step‑by‑step walkthrough of buffer‑overflow exploitation on Linux, detailed shellcode construction, and practical defensive measures using Libsafe, while highlighting the limits of both attack and defense strategies.