How Integer Overflows Threaten C Programs and How to Detect Them Safely

Integer overflow is a long‑standing issue in C programming that is often overlooked, yet it can cause buffer overflows and enable various attacks.

What is integer overflow

For unsigned types the C standard defines overflow as a modulo operation with 2^(8*sizeof(type)). For example:

unsigned char x = 0xff;
printf("%d
", ++x);

The program prints 0 because 0xff+1 equals 256, which modulo 2^8 yields 0.

For signed types the standard specifies undefined behavior, so the compiler may handle it arbitrarily. Example:

signed char x = 0x7f;
printf("%d
", ++x);

This prints -128 because the result wraps to the most‑negative value.

Signed overflow is not always a negative number; consider multiplication:

signed char x = 0x7f;
signed char y = 0x05;
signed char r = x * y;
printf("%d
", r);

The output is 123.

Risks of integer overflow

Example 1: overflow causing an infinite loop

short len = 0;
while(len < MAX_LEN) {
    len += readFromInput(fd, buf);
    buf += len;
}

If MAX_LEN exceeds the range of short (e.g., 32767), len can become negative and the loop never terminates.

Example 2: overflow during type conversion

int copy_something(char *buf, int len) {
    #define MAX_LEN 256
    char mybuf[MAX_LEN];
    if(len > MAX_LEN) return -1;
    return memcpy(mybuf, buf, len);
}

Here len is signed while memcpy expects an unsigned size_t. A negative len is converted to a large unsigned value, causing a buffer overflow.

Example 3: heap overflow in OpenSSH

nresp = packet_get_int();
if(nresp > 0) {
    response = xmalloc(nresp * sizeof(char*));
    for(i = 0; i < nresp; i++)
        response[i] = packet_get_string(NULL);
}

By supplying a crafted nresp (e.g., 0x40000001), the multiplication overflows, the allocation becomes too small, and the subsequent loop overwrites arbitrary memory.

Example 4: buffer overflow via unchecked length

int func(char *buf1, unsigned int len1, char *buf2, unsigned int len2) {
    char mybuf[256];
    if((len1 + len2) > 256) return -1;
    memcpy(mybuf, buf1, len1);
    memcpy(mybuf + len1, buf2, len2);
    do_some_stuff(mybuf);
    return 0;
}

If len1 + len2 overflows, the condition may be false and the copies write past mybuf.

Example 5: size_t overflow in reverse loops

for(int i = strlen(s)-1; i >= 0; i--) { ... }

When strlen(s) is 0, i becomes a huge unsigned value, leading to out‑of‑bounds access.

Compiler behavior

Compilers assume programs do not overflow under optimization flags -O2 / -O3 and may eliminate overflow checks.

Optimization example

int len;
char *data;
if(data + len < data) {
    printf("invalid len
");
    exit(-1);
}

With -O2 the entire if block disappears.

Casting the pointer to an unsigned integer type prevents the optimizer from removing the check:

if((uintptr_t)data + len < (uintptr_t)data) {
    ...
}

C99 §6.5.6 defines pointer arithmetic overflow as undefined behavior, giving the compiler freedom to handle it arbitrarily.

Correct detection of integer overflow

Checks must be performed before the arithmetic that could overflow, and signed‑to‑unsigned promotions should be avoided.

void foo(int m, int n) {
    size_t s = 0;
    if(m > 0 && n > 0 && (UINT_MAX - m < n)) {
        // error handling
        return;
    }
    s = (size_t)m + (size_t)n;
}

Use UINT_MAX (or INT_MAX) rather than SIZE_MAX for portable checks.

Overflow in binary search

int binary_search(int a[], int len, int key) {
    int low = 0;
    int high = len - 1;
    while(low <= high) {
        int mid = low + (high - low) / 2; // safe
        if(a[mid] == key) return mid;
        if(key < a[mid]) high = mid - 1; else low = mid + 1;
    }
    return -1;
}

Using low + high can overflow; the rewritten formula avoids it.

Checking addition overflow

void f(signed int a, signed int b) {
    if((b > 0 && a > INT_MAX - b) || (b < 0 && a < INT_MIN - b)) {
        // handle error
        return;
    }
    int sum = a + b;
}

Checking multiplication overflow

void func(signed int a, signed int b) {
    if(a > 0) {
        if(b > 0) {
            if(a > INT_MAX / b) { /* error */ }
        } else {
            if(b < INT_MIN / a) { /* error */ }
        }
    } else {
        if(b > 0) {
            if(a < INT_MIN / b) { /* error */ }
        } else {
            if(a != 0 && b < INT_MAX / a) { /* error */ }
        }
    }
    int result = a * b;
}

These patterns follow the guidance in "INT32‑C. Ensure that operations on signed integers do not result in overflow" and the Apple Secure Coding Guidelines.