From Prompt to Graph: Understanding the Five‑Layer Agent Architecture

Five Layers and Their Boundaries

Prompt : defines a single task’s goal, scope, constraints, output, and completion criteria.

Context : supplies the minimal sufficient work set the model needs at the moment—facts, code, state, version, and timeliness.

Harness : describes the runtime environment outside the model—tools, permissions, budget, logging, and rollback mechanisms.

Loop : decides, after new evidence, whether to continue, stop, roll back, or take an alternative path.

Graph : records stable relationships, dependencies, branches, merges, approvals, and recoveries across multiple work units.

These layers are not a maturity ladder; a simple task may only need the first three, while a long‑running, well‑defined task often succeeds with a single Loop. Even a complete Graph cannot compensate for a badly written Prompt or stale Context.

How a Task Traverses the Five Layers

Consider the example “fix occasional duplicate payment callbacks”. The Prompt must specify the target service, immutable accounting fields, and the test that signals success. The Context then provides the current branch code, the latest failure log, the callback protocol, the database schema, and reproducibility steps.

During execution, the Harness determines which tools the model may invoke, the isolation environment, budget consumption, write‑operation confirmations, and result validation. This is where the model meets the external world.

If new logs appear, the original hypothesis may be invalidated; failed tests may require the Loop to incorporate feedback, stop conditions, or a fallback path.

When the task involves reproducibility, code changes, CI, approval, canary rollout, and rollback, the Graph captures the stable dependencies and failure paths, making the system aware of where it can retry, what requires human approval, and how to resume.

Layer Interaction and Nesting

Conceptually the layers expand outward, but at runtime they repeatedly nest. A Graph node can contain a Loop; each Loop iteration reassembles a Context, calls Harness tools, and issues a more concrete Prompt. The layers act simultaneously at different scales rather than sequentially.

Extending the Model: From Simple Tasks to Complex Workflows

Moving outward adds responsibility. Prompt and Context affect a single model decision and suffer from mis‑answers, missing constraints, or outdated versions. Harness and Loop introduce real costs—tool usage, budget, logs, and potential large‑scale failures.

Graph adds cross‑process, cross‑Agent, cross‑team coordination, shifting focus from “which model to call next” to questions of state ownership, failure recovery, side‑effect duplication, and approval responsibility.

These concerns are not new: Kubernetes controllers implement a control Loop, and GitHub Actions use needs to express job dependencies, forming a task graph. Agent engineering adds the ability for nodes to adjust routes based on live evidence.

AutoResearch: A Concrete Five‑Layer Example

Andrej Karpathy’s AutoResearch demonstrates a compact loop that modifies a tiny language‑model training script, runs a fixed‑duration training, reads a validation metric, and decides whether to keep or revert the change.

Propose idea → modify train.py → commit → train → read val_bpb → keep or revert → next round

Mapping the five layers:

Prompt : program.md describes the goal, editable range, logging, and selection criteria (minimize val_bpb).

Context : the Agent reads the task description, current code, Git state, and results.tsv experiment history.

Harness : only train.py, prepare.py, evaluation functions, data, and environment are mutable; each training run lasts ~5 minutes and logs results.

Loop : if the metric improves, keep the change; if it degrades or crashes, revert; after repeated failures, abandon the idea.

Graph : a full system graph is unnecessary unless multiple research directions need parallel comparison, merging, or partial recovery.

Karpathy reports roughly 700 autonomous modifications over two days, yielding ~11 % reduction in “Time to GPT‑2” for the nanochat benchmark. The author cautions that this improvement is specific to the codebase and metric and does not imply a universal 11 % productivity gain.

When to Use Loop vs. Graph

Loop excels at paths that cannot be predetermined—online fault investigation, where the next step depends on logs or alerts. Graph excels at stable relationships—once a reproducible fix passes CI, the Graph records the required approvals, canary rollout, and rollback conditions.

Three practical questions help decide:

Is the relationship stable? Can dependencies, branches, and merge conditions be written before execution?

Is failure expensive? Does interruption require partial recovery, and would retries cause duplicate charges or writes?

Does the process need accountability? Are there audits, approvals, budgets, or cross‑team handoffs?

If the signals are weak, a Loop is simpler; if any signal becomes strong, extracting the relationship into a Graph is worthwhile.

Practical Guidance for Building Robust Agent Systems

Record the real execution route: which tools were used, where failures occurred, which steps are stable, and which paths vary.

Move state out of the dialogue: store task ID, input version, current stage, key artifacts, budget, and last error so the system knows where it stopped.

task_id: repair-20260731-001
input_version: git:8f3a2c1
stage: reproduce
status: running
attempt: 2
budget_remaining_seconds: 1800
artifact_uri: runs/repair-001/
last_error: null

Place validators at explicit gates: tests, metrics, data schemas, code diffs, and permission rules should be checked in fixed locations; human confirmation may be needed for subjective judgments.

Isolate side‑effecting actions (publish, write DB, payment, notification) as separate nodes with minimal permissions, idempotent keys, audit logs, and compensation logic.

Parallelize only after confirming independence: clear data dependencies, write sets, resource limits, and merge rules must be defined.

Following these steps usually adds only a few extra states and edges, yet the resulting graph can pause, partially recover, and explain why the system reached a given point.

Reviewing an Agent Graph: Eight Questions

Before adopting a Graph framework, answer these eight items:

State : Who owns the task state? Is there a task ID, versioned state structure, current stage, remaining budget, and artifact location?

Node : Are input/output structures explicit? Which tools and permissions are allowed? How are timeout, cancellation, and retries expressed?

Edge : Why must B wait for A? Is it data dependency, control dependency, or just visual ordering?

Acceptance : What evidence proves success? Which checks can be automated and which need human judgment?

Failure : Which errors merit retries and how many? After repeated failure, should the system downgrade, roll back, skip, or hand over to a human?

Side‑effects : Do writes, deployments, payments, or notifications have idempotent keys and audit records? How are partial successes compensated?

Recovery : Where are checkpoints stored? Do they contain complete, verifiable artifacts or just a “completed” flag?

Observability : Can you see node latency, cost, failure rate, retry count, and human wait time?

If many answers are “Agent will handle it”, the graph is likely still at a schematic level; adding concrete state, evidence, and failure paths is more valuable than further node decomposition.

Final Thoughts

The five‑layer model clarifies responsibilities: Prompt states the work, Context provides the current facts, Harness controls the execution environment, Loop explores and converges, and Graph records stable cross‑unit relationships. When building systems, write familiar paths as edges, leave unknown exploration to Loops, and protect high‑risk decisions with explicit, accountable gates.

References

Peter Steinberger – “Are we still talking loops or did we shift to graphs yet?”

David Khourshid – “State machines in 2 minutes”

Andrej Karpathy – AutoResearch repository and summary of ~700 modifications

LangChain – “3 Years of Graph Engineering with LangGraph”

LangChain – “The Art of Loop Engineering”

Anthropic – “Building Effective AI Agents”

GitHub Docs – Workflow syntax for GitHub Actions

Kubernetes Docs – Controllers