Uncovering a Hidden Compiler Backdoor on a 3B2 System – A Day‑by‑Day Debugging Saga

Day 1

I was hired by a psychologist to repair a C program that asked users 50 questions, performed calculations, and produced a score. The software ran on a DEC 3B2 computer (see https://en.wikipedia.org/wiki/3B_series_computers#3B2). The code was deliberately obfuscated: all 15 source files contained three one‑line functions each, with three‑letter variable names. After reformatting the code for readability, I found it used the curses library to display prompts. Deleting a problematic mvprintw() call seemed to fix the overwritten message, but the bug reappeared on the next run.

Day 2

Scanning the hard drive yielded no useful clues. The mysterious string appeared to be either encrypted or hidden inside a library, so I began searching every library for the text, a time‑consuming process.

Day 3

With no results, I traced #include files from one header to another, suspecting the code lived in a compiled library. The university’s computer department was notified because the machine was a shared research workstation, and they were skeptical of my claims.

Days 4‑6

The hardest part was convincing the campus staff that a problem existed. Mark, the Unix administrator, eventually let me take over. We recompiled all standard libraries, but after each compilation the original 15 source files and the overwritten strings re‑materialised, as if a hidden process restored them. I discovered a hidden program that copied the original source back whenever the code was compiled. After copying the source tree, setting it read‑only, and recompiling, the same restoration occurred.

Day 7

Determined not to be defeated, I declared I would either compile the malicious code myself or abandon the task, demanding only research time from Dr. Phelps.

Days 8‑14

I suspected the curses library had been altered. I compiled the library to assembly, spent six days studying the 3B2 assembly manuals, and tried to locate the offending code, but everything appeared normal.

Day 15

The breakthrough came when I realized the compiler itself was compromised. The malicious compiler performed the following steps:

1) Scan every fopen() call for the questionnaire file; if found, then
2) When compiling a specific program, overwrite the 15 source files in the current directory.
3) Compile the program using those overwritten files and link with -o output.

In other words, the compiler injected code into Dr. Phelps’s program during compilation. AT&T technicians later supplied a clean compiler and linker source, which we rebuilt, removing the harmful code. However, other hidden modifications persisted in the binary compiler, re‑injecting changes each time it compiled. We finally used a binary compiler from another 3B2 machine to break the cycle.

We also discovered that if /sbin/login was compiled with the tainted toolchain, it created a backdoor allowing login as root with a specific password, exposing the system via modem and Tymnet. The incident eventually attracted the attention of the university’s computer center.