Can Large Language Models Rebuild Complex Systems? ProgramBench’s Harsh Verdict

ProgramBench Overview

Stanford NLP’s ProgramBench benchmark evaluates how well mainstream large language models can reconstruct complex software projects from only an executable binary.

The benchmark uses 200 complete code repositories, including SQLite, FFmpeg, and a PHP compiler. For each case the model receives only the compiled executable and must regenerate the full source code.

Methodology

Test set contains 248,000 concrete cases covering the full development pipeline: design, implementation, building, and debugging.

Projects were deliberately chosen for deep domain knowledge requirements; for example, rebuilding the PHP compiler involves syntax parsing and low‑level memory management.

Models operate in a fully network‑isolated environment with no reference or starter code, forcing reverse‑engineering from the binary.

Results

All models achieve a 0 % actual solve rate (complete, correct reconstruction).

Claude Opus 4.7 attains the highest “almost‑solve” rate at 3 % (occasional generation of near‑usable code fragments).

Claude Opus 4.6 records 2.5 % almost‑solve, Claude Sonnet 4.6 records 1 %.

GPT‑5.4 and Gemini 3.1 Pro score 0 % on both actual and almost‑solve metrics.

Analysis

Across models a consistent pattern emerges: they can generate plausible interface definitions, but implementation of key algorithms quickly collapses. This indicates stronger capability for mimicking surface‑level code structure than for understanding underlying logic.

The harsh test conditions—network isolation and absence of starter code—make the task substantially harder than typical code‑assistant usage, requiring full reverse engineering of system design from binaries.

These findings reassure architects and low‑level programmers that current models handle simple algorithmic tasks and small tools reasonably well, but they remain far from capable of authentic system‑level codebase reconstruction.

Full report: https://programbench.com