Can LLMs Engineer Their Own Infrastructure? A Deep Dive into Φ‑Bench’s Assessment

Motivation

Large language models (LLMs) have progressed from code generation to complex reasoning and software‑development assistance. Φ‑Bench (Frontier AI Infrastructure Benchmark) asks whether LLMs can also engineer the infrastructure that runs them.

Benchmark construction

Tasks were mined from four years of top‑tier systems papers and open‑source infra repositories. From millions of papers, issues and pull‑requests the team extracted >10,000 candidate tasks, automatically filtered to >4,000 high‑value candidates, and then domain experts manually reviewed and refined the set to 85 real, high‑difficulty tasks . A hierarchical taxonomy with 410 fine‑grained tags, 62 middle‑level topics and 9 high‑level topics ensures systematic coverage of modern LLM‑infra challenges.

Task categories

Kernel Function Completion (KFC) : Implement and optimise a single operator given a fixed interface and I/O semantics, then verify correctness and improve performance.

Long‑Horizon Implementation (LHI) : Given an issue‑style requirement, understand a large codebase, locate relevant modules, modify multiple files and achieve end‑to‑end functionality through iterative testing and debugging.

End‑to‑End Optimization (E2EO) : Optimise a full system workload under specified objectives and constraints without a prescribed solution path. The model must analyse bottlenecks, devise optimisation strategies and iteratively modify code across modules and layers.

Evaluation results

Frontier models were evaluated on the 85 tasks. Overall scores (higher is better) were:

Claude Opus 5 – 36.53

Kimi K3 – 28.12

Qwen 3.8 Max – 27.73

Model strengths differed across infra dimensions:

Claude Opus 5 led in most topics.

Kimi K3 excelled in Inference & Serving and System Optimisation.

GLM 5.2 performed best in System Assurance.

Even the best model scored only 5.4 on Hardware & Edge tasks, indicating a substantial gap in low‑level system optimisation.

Continuous optimisation (E2EO) ability

In the multi‑round E2EO task, models received experimental feedback and were required to iteratively improve their solutions. Claude Opus 5 started from a relatively good baseline and consistently refined it. Qwen 3.8 Max and Kimi K3 began from lower baselines but quickly caught up through rapid iteration. Some models stalled and failed to make further progress, demonstrating that infra optimisation is a long‑term engineering process rather than a one‑shot code generation problem.

Reasoning‑budget impact

Increasing the inference (reasoning) budget generally improved performance, but gains were non‑linear and varied markedly between models, revealing divergent dependencies on additional compute.

Error‑pattern analysis

Errors observed during full‑trajectory optimisation were categorised into four groups: Python runtime errors, CUDA execution errors, Triton/MLIR/CUDA compilation errors, and tensor‑shape mismatches. High‑scoring models produced more errors because they explored more complex optimisation directions; lower‑scoring models made fewer changes and thus encountered fewer errors. Claude Opus 5’s errors were predominantly CUDA execution errors, indicating it moved quickly beyond “code runs” to harder system‑level challenges.

Task openness and model challenge

Performance degraded from KFC to LHI to E2EO, confirming that understanding large codebases and coordinating multi‑file changes remains the core difficulty for LLMs.

Anti‑cheating mechanisms

Φ‑Bench employed soft network isolation and prompt constraints to prevent hacking. A rule‑based detector flagged only three attempts by DeepSeek V4 Pro to access external PyTorch resources; manual review confirmed these did not constitute actual cheating. The mechanisms effectively ensured trustworthy evaluation.

Characteristics of strong infra models

Planned action : Form low‑cost hypotheses and verify them before committing to code changes, reducing iteration cost.

Experimental design : Design controlled experiments that isolate variables and noise, extracting maximal information from each trial.

Careful interpretation : Avoid naïve attribution of performance changes; actively rule out confounding factors to prevent misguided optimisation directions.

Significance of Φ‑Bench

Traditional Recursive Self‑Improvement (RSI) research focuses on better data, algorithms and architectures. Φ‑Bench adds a new dimension: assessing whether models can autonomously improve the infrastructure that runs them. Success would enable AI systems not only to consume compute resources but also to participate in building the next generation of AI infrastructure.

文章链接：https://github.com/one2piece2hello/faibench_Frontier_InfraBench/blob/main/faibench.pdf
项目网站：https://faibench.org/
github：https://github.com/one2piece2hello/faibench_Frontier_InfraBench
hugging face：https://huggingface.co/datasets/faibench-Frontier-Infra-Bench/faibench_Frontier_Infra_Bench