Skip to content
LLMs for AI Infrastructure: Lifecycle Survey Shows Gains Not Comparable, Handoffs Missing

LLMs for AI Infrastructure: Lifecycle Survey Shows Gains Not Comparable, Handoffs Missing

This survey reviews how large language models assist across the AI infrastructure lifecycle—from chip design and operator optimization to cluster orchestration, data management, and energy efficiency—highlighting that reported gains often use incompatible metrics and baselines, and cross-stage validation remains a critical gap.

Data Party THU
Data Party THU
Data Party THU

Introduction

Large language models (LLMs) not only consume compute infrastructure but increasingly participate in improving it: generating chip designs, rewriting GPU operators, optimizing compilation strategies, configuring training parallelism, diagnosing cluster failures, and managing data and energy. However, these efforts span hardware, systems, HPC, and ML, and claimed "multi-fold improvements" often differ in metrics, baselines, or environments. The survey Large Language Models for AI Infrastructure (arXiv:2611.19061, November 2026) by Bolin Shen, Shaoyi Huang, and Yushun Dong connects this research across the full lifecycle. The paper spans 54 pages and organizes contributions around three contributions: cross-lifecycle synthesis, a lifecycle-aligned taxonomy, and quantitative synthesis preserving original metrics and conditions.

The central thesis: engineering gains depend on candidate generation, verification feedback, baseline strength, and final workload impact. Passing compilation does not guarantee correctness; operator speedup does not translate to proportional training acceleration; better diagnostic reports do not equal fixed failures.

Background and Problem Formulation

AI Infrastructure Lifecycle

The paper defines seven stages: accelerator design, operator development, compilation, deployment & serving, runtime operations, data supply, and power management. The first three build compute artifacts; the middle two run workloads; the last two supply resources. Stage handoffs (e.g., generated operator entering inference engine, energy measurements feeding back into hardware targets) are as important as single-stage optimizations, yet most methods remain confined to single stages, with particularly sparse validation for accelerator-to-operator and data-to-power handoffs.

Figure 1: Mapping of seven stages to three activity categories. Solid lines indicate existing cross-stage evaluation evidence; dashed lines indicate missing end-to-end validation. Not all stages form an automatically executable closed loop. Source: original Figure 2, page 4.
Figure 1: Mapping of seven stages to three activity categories. Solid lines indicate existing cross-stage evaluation evidence; dashed lines indicate missing end-to-end validation. Not all stages form an automatically executable closed loop. Source: original Figure 2, page 4.

Common Task Formalization

Tasks are formalized via inputs (specs, code, telemetry, constraints), candidate actions (programs, config changes, diagnoses), evaluation environments (tools, hardware, runtime state), feedback mechanisms, and target costs. The goal is to optimize cost or performance under functional, safety, and service requirements. However, system-visible tests and simulators are only proxies; test coverage gaps, loose numerical tolerances, simulator errors, or judge biases can decouple optimized scores from true engineering value. Thus the framework is unified, but correctness standards and optimization functions differ across tasks.

Landscape of LLM-Assisted AI Infrastructure Engineering

Taxonomy Overview

Methods are grouped by primary managed object into three groups of eight categories:

Artifact group : hardware design, operator engineering, HPC software.

Runtime group : system integration, production operations, workload orchestration.

Resource group : data management, energy optimization.

Model adaptation style, feedback type, and lifecycle stage are retained as attributes, not primary classifiers. A work may span categories but is assigned by core contribution.

Figure 2: Three groups and eight categories by primary managed object. Model adaptation, feedback type, and lifecycle stage are attributes, not primary classification bases. Source: original Figure 3, page 6.
Figure 2: Three groups and eight categories by primary managed object. Model adaptation, feedback type, and lifecycle stage are attributes, not primary classification bases. Source: original Figure 3, page 6.

Compute Artifact Automation

Hardware Design

LLMs generate or repair RTL, HLS code, and circuit designs from natural language or existing designs, verified via compilation, simulation, formal methods, and synthesis flows for functionality, power, performance, area. Key approaches: domain fine-tuning, compilation-diagnostic-driven repair (e.g., RTLFixer ), symbolic tools with formal verification ( CktFormalizer ), population search, and multi-agent design flows ( COEVO co-searches functionality and multi-objective metrics). Critical limitation: passing tests ≠ meeting design contracts; syntax correctness, simulation pass, and physical design quality are distinct outcomes. Search budget, prompt configuration, and benchmark version affect success rates. Direct evaluation targeting training/inference accelerators remains rarer than general circuit evaluation.

Operator Engineering

LLMs generate or optimize device code (CUDA, Triton, HIP) with numerical verification and execution-time feedback. KernelBench -style benchmarks require candidates to pass correctness tests and exceed a speedup threshold; candidate count increases chance of success, so search budget must be reported. Approaches evolve from simple execution feedback to performance profiling, structured transformations, RL, evolutionary search, and expert-library composition. AutoKernel targets whole-model bottlenecks; Ave uses dataflow invariants to constrain edits; CUDA Agent and Dr. Kernel explore training-time feedback; CUCo explores compute-communication co-optimization. Correctness tests do not guarantee correctness across all inputs, precisions, or numerical conditions; fixes may regress to slower but easier-to-pass implementations. Cross-hardware porting requires re-verification of numerics and performance, not just recompilation.

HPC Software

Covers compilation optimization, DL frameworks, parallel program translation, and concurrency bug handling. LLMs propose compiler pass sequences, synthesize inlining or register-allocation heuristics, generate OpenACC directives, translate OpenMP/CUDA, assist MPI error localization. LLM Compiler , AwareCompiler , Magellan explore compilation decisions and heuristic search; PassNet targets graph compilation; FuzzGPT and AlignGuard address framework bugs. Repo-level parallelization must handle build systems, cross-file dependencies, deadlocks, OOM — single-function results are insufficient. Amdahl's law limits local gains by their fraction in the whole task; communication, synchronization, and memory access further affect realized speedup. Instruction-count reduction, binary size reduction, and training time reduction must be evaluated separately.

Runtime Control

System Integration

Connects research artifacts to executable environments: dependency installation, build recipes, containers, Kubernetes configs, service interfaces. DeployBench verifies paper-specified experiments in fresh VMs; SpackIt generates build recipes from repos and install diagnostics; GenKubeSec handles config errors. Config syntax validity ≠ service availability; deployment success ≠ long-term stability, security, or load performance. Collected integration systems have not yet evaluated full training/serving stack deployment — a key gap between general software deployment and AI infrastructure practice.

Production Operations

Models ingest logs, metrics, traces, events to output root causes, explanations, or restart/config suggestions. Retrieval, structured evidence, and tool use improve information organization, but diagnostic label accuracy ≠ fix success rate. Evaluation must distinguish offline logs, replayable failures, and real production incidents. Allowing autonomous state changes introduces downtime and misoperation costs; verification must check how recommendations translate to correct interventions and actual service recovery.

Workload Orchestration

Decides how training/serving use shared GPUs, memory, interconnect: parallelism, sharding, routing, batching, KV cache, expert placement, job scheduling, instance selection. ASAP , OptiCo , Primus study training config; Glia explores serving policies; CacheCraft studies cache eviction; SchedMate and AlphaEvolve address scheduling/heuristics. LLMs often propose candidates; simulators, performance projection engines, and real tests determine search quality. Evidence from simulators, offline frameworks, small real clusters, and production platforms cannot be conflated; identical speedup ratios may correspond to vastly different absolute savings at different workload scales.

Resource Optimization

Data Management

Includes pipelines, data contracts, catalog discovery, structured extraction, analysis agents. DataFlow targets training data prep; Evaporate turns per-document LLM calls into reusable extraction programs; SANA studies tool support and analysis workflows. Resource gains come not only from smaller models but from converting one-off LLM calls into reusable programs. Extraction quality, data coverage, and call cost must be jointly verified; reduced call count does not guarantee lower total workflow cost.

Energy Optimization

Software side: measurement feedback, energy-aware hints, candidate screening to reduce execution energy. Hardware side: clock gating, RTL rewriting, multi-objective search for power. Facility side: cooling load prediction. Energy conclusions depend on device, workload, measurement boundary, and reference implementation. More accurate prediction ≠ lower datacenter power; reduced program energy may not cover search and inference overhead. Carbon estimates depend on runtime and local grid carbon intensity.

Benchmarks and Quantitative Synthesis

Benchmarks, Metrics, and Evidence Extraction

This chapter compiles original results without re-running all systems. Recoverable metric definitions, values, baselines, and conditions are preserved; derived results are marked. Comparison dimensions: functional correctness, execution performance, throughput, resource/energy, ops outcomes, optimization cost. Cross-comparison only when conditions permit; no collapsing of pass rates, speedups, diagnostic quality, energy reduction into a single score.

Compute Artifact Evaluation

Hardware : Syntax vs. functional success rates can diverge; fine-tuning, feedback, and search budget co-vary, so highest success rate alone does not indicate model superiority. Power, performance, area must be compared under functional validity.

Figure 3: RTL generation results across original studies showing syntax vs. functional success, different test splits, and search budgets. Each panel retains its own protocol and metrics; not a unified leaderboard. Source: original Figure 4, page 23.
Figure 3: RTL generation results across original studies showing syntax vs. functional success, different test splits, and search budgets. Each panel retains its own protocol and metrics; not a unified leaderboard. Source: original Figure 4, page 23.

Operators : Passing tests and beating optimized references are distinct goals. Same-budget iterative feedback helps some models but hurts others; ablation does not support "more performance feedback always helps."

Figure 4: Pre/post feedback changes, functional and speed metrics under common reference, and speedup relative to each baseline. Large gains over unoptimized implementations vs. beating optimizing compilers represent different engineering value. Source: original Figure 5, page 24.
Figure 4: Pre/post feedback changes, functional and speed metrics under common reference, and speedup relative to each baseline. Large gains over unoptimized implementations vs. beating optimizing compilers represent different engineering value. Source: original Figure 5, page 24.

HPC Software : Reference strength matters. Gains over weak compiler baselines may shrink to marginal improvements under strong baselines. Textual similarity to reference instructions is weaker evidence than real parallel execution and scaling.

Runtime Evaluation

System Integration : Must record environment, dependencies, permissions, timeouts, success criteria. Operations : TraceBench uses fixed LLM judge to rank reports; accuracy, utility, explainability may not align; these are judge-assigned rankings, not observed recovery outcomes. Workload Orchestration best illustrates condition sensitivity. Glia reports 2.2× avg request completion time reduction vs. minimal-load routing in simulator; another real testbed reports slowdown — cannot treat as same speedup. In production, AlphaEvolve 's Borg heuristic recovers ~0.7% fleet-wide compute; small fraction can still mean large absolute resources.

Figure 5: Original study gains and search times across environments. Point shapes distinguish simulation, offline, real cluster, production; each row has independent metrics and baselines, so cannot be ranked by horizontal axis. Source: original Figure 7, page 29.
Figure 5: Original study gains and search times across environments. Point shapes distinguish simulation, offline, real cluster, production; each row has independent metrics and baselines, so cannot be ranked by horizontal axis. Source: original Figure 7, page 29.

Notable controlled comparisons:

OptiCo achieves 101.84% of best expert MFU (i.e., 1.84% relative improvement), not 101.84% improvement.

Primus 's 27% measured gain over public reference includes BF16→FP8 precision change; not fully attributable to agent.

Primus ablation: same agent with more accurate, benchmark-calibrated evaluator yields better projected throughput; with weaker analytical evaluator, exhaustive search beats LLM agent. Supports "propose cross-dimensional candidates under reliable evaluation," not "LLM replaces performance evaluator."

Gains decay from local to global: AlphaEvolve reports 23% avg operator speedup but ~1% overall training time reduction. Search cost must include simulator/projection engine construction, not just single-call cost.

Resource Evaluation

Data Management : Evaporate improves average quality and significantly reduces tokens via reusable extraction code, but quality drops on Enron data; direct prompting vs. code solutions differ in sample coverage. Average gain does not obviate per-source verification. SANA 's Ideal tool configuration contains privileged information — an upper bound, not deployable solution; real complex workflows may not approach this bound, indicating need to separately diagnose planning, source selection, analysis bottlenecks.

Energy : Similar device averages can mask per-application variance. In Table 15, LASSI-EE shows similar average savings on MI100 and A100, but for entropy application with candidate budget 5, expected savings differ by 81.9 percentage points. This is not average energy change and cannot indicate universally better platform. Another study: model-generated code vs. human code consumes 18.5% more energy on server, 24.3% less on PC, 2.3% more on Raspberry Pi (9 EvoEval problems, specific measurement conditions) — not a general conclusion, but explicitly shows cross-device transfer requires re-measurement.

Figure 6: Top table compares human vs. model code energy for specified programs and prompts; bottom table lists candidate budgets and device differences. Gains must be explained with application, hardware, and reference implementation. Source: original Table 15, page 31.
Figure 6: Top table compares human vs. model code energy for specified programs and prompts; bottom table lists candidate budgets and device differences. Gains must be explained with application, hardware, and reference implementation. Source: original Table 15, page 31.

Cross-Category Synthesis

Three principles:

Feedback is a design component; its benefit depends on model and task, should be validated under fixed budget.

Evaluation endpoint must match output role: artifacts → functional + real execution; diagnostics → correct intervention + recovery; orchestration → representative workload; resource actions → direct savings under maintained quality.

Optimization cost should be amortized by reuse scope. One search yielding an operator, compiler heuristic, or extraction function has different amortized value depending on reuse count. Unmeasured connections from local scores to end-to-end gains should be future experiments, not replaced by higher local scores.

Related Work

Specialized surveys in hardware, operators, compilation, parallel programming provide technical depth; infrastructure-as-code and AIOps explain deployment/failure workflows; distributed training and efficient serving surveys describe the design space of optimized systems. Key difference: studying training/serving systems ≠ studying LLMs as engineering agents. Data and energy-efficient code surveys cover only parts of lifecycle; circuit power and facility prediction have different measurement requirements. This survey's unique value: placing "what the model manages, what feedback it receives, how it's validated" in one lifecycle, using original evidence to show which conclusions are comparable and which must retain differences.

Conclusion

LLM-assisted methods now touch every AI infrastructure stage, but training/serving-specific problems and cross-stage handoffs remain understudied. Checkpoint recovery, slow-node handling, quantization selection, data loading, job power control — within this survey's scope — lack sufficient research. Clearest value appears where executable, measurable feedback exists; production credibility demands stronger baselines, evaluator fidelity, representative workloads, real deployment validation. This paper is neither proof of autonomous infrastructure experts nor a cross-study unified re-run.

When reading any "LLM makes system faster" result, ask four questions: Is it correct? Faster than what? Where measured? Does it improve final workload after accounting for optimization cost? From local generation to cross-hardware/software/runtime co-design, these evidences must be connected segment by segment.

Paper: Large Language Models for AI Infrastructure , Bolin Shen, Shaoyi Huang, Yushun Dong. arXiv:2611.19061v1, 14 Nov 2026. https://arxiv.org/abs/2611.19061 |

https://arxiv.org/pdf/2611.19061
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Large Language ModelsBenchmarkingAI infrastructureEnergy EfficiencyHardware DesignOperator OptimizationCluster OrchestrationCross-Stage Validation
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.