Understanding Stack Pointer Corruption When Mixing RSP and ESP in 64‑bit Windows: A Reverse‑Engineering Case Study

A company replaced an old DLL with a new version in a Windows application; the program runs fine on Windows 7 but crashes immediately on Windows 10 with a flash‑out error.

Disassembly of the failing function shows a prologue that allocates stack space with sub rsp, 298h (64‑bit stack pointer) and an epilogue that restores the stack with add esp, 298h , mixing the 64‑bit rsp and the 32‑bit esp registers.

On x86‑64 CPUs, any write to a 32‑bit register automatically zero‑extends the upper 32 bits of the corresponding 64‑bit register. Therefore, executing add esp, 298h clears the high 32 bits of rsp , turning the stack pointer into a “wild” address on systems where those high bits are non‑zero (Windows 10). On Windows 7 the high 32 bits were already zero, so the bug was invisible.

Windbg analysis confirms that after the add esp, 298 instruction the high 32 bits of rsp become zero, leading to a 0xC0000005 access‑violation crash. The article includes screenshots of the crash dump and register state.

Microsoft documentation and the Intel x86‑64 manual are cited, stating that writes to 32‑bit registers zero‑extend, while reads do not, and that certain 8‑bit registers cannot be used with the REX prefix in 64‑bit mode.

The root cause is thus the inconsistent use of rsp and esp in the same function; fixing the code to use matching 64‑bit registers (or consistent 32‑bit code) resolves the crash.

Additional images from the original post illustrate the disassembly, crash dump, and register values.