Why More Automation Means More Human Judgment in Loop Engineering

What is Loop Engineering?

Loop Engineering has recently become a hot topic. Influential voices such as Peter Steinberger and Boris Cherny note that the emphasis is moving from single‑round prompt‑based coding agents to designing loops that guide agents like Claude in deciding the next steps.

Beyond Prompt Engineering

The key insight is that loops do not eliminate prompts; they embed prompts inside a feedback system that discovers work, assigns tasks, executes, validates, records state, and then decides whether to continue, stop, or hand over to a human.

From Harness to Loop

Earlier discussions about Harness Engineering described a workbench with tools, context, state, tests, and permissions. Loop Engineering adds the requirement that the workbench can periodically “wake up,” discover new problems, handle them, and leave evidence.

What Makes a Loop Viable?

Automatic trigger

Isolated worktree or temporary branch

Process assets (skills, templates, verification scripts)

External connections (plugins, CLI, APIs)

Independent verification

State memory (e.g., a plan.md file)

When trying a loop in a team, start with low‑risk scenarios such as factual verification of technical articles, CI failure triage, or configuration drift checks.

Loop Architecture

Trigger entry: Who can start the loop, how often, and where the input comes from.

Execution sandbox: The worktree, branch, or temporary environment where the agent runs, with rollback capability.

Acceptance outlet: Tests, logs, rules, and manual review that decide if the result is acceptable.

State ledger: Where each attempt, evidence, failure reason, and next step are written.

Closed‑Loop vs. Open‑Loop

A closed loop has a clear goal, bounded actions, objective feedback, and verifiable stop conditions (e.g., CI triage, fact‑checking). An open loop lets the agent explore autonomously, which increases risk because budgets, goals, and verification become harder to control.

Validation Checklist

Before adopting a loop, evaluate the following:

Input: Stable sources such as logs, issues, test reports.

Output: Structured artifacts like classification tables, candidate PRs, evidence lists.

Verification: Automated tests, linters, link checks, reproducible commands.

Permission: Default read‑only, writes go through isolated branches.

Stop conditions: Budget exhausted, insufficient evidence, or need for manual decision.

If at least two items fall into the “delay” column, first improve testing, state tracking, and boundaries before automating the loop.

Concrete Examples

Technical article fact‑checking: An agent extracts factual statements, then cross‑checks them against official docs, code repositories, papers, or issues, marking each as confirmed, insufficient source, possibly outdated, speculative, or delete‑worthy.

CI failure triage: Each day the loop reads recent CI failures, categorises them (environment issue, flaky test, recent commit, known problem), and generates a table. If a failure can be linked to a specific commit, the loop opens an isolated worktree for a minimal fix; otherwise it routes the case to a human.

Budget Considerations

Loops can quickly become costly because each round may reread context, call tools, generate plans, and run verification. Sub‑agents amplify this cost. Therefore, define limits such as maximum runtime, maximum number of branches, and token budget in a task card, e.g.:

Loop name: Daily CI triage
Trigger: 09:30 each day
Input: last 24 h failed jobs, last 20 commits
Max runtime: 30 min
Max branches: 5 failure clusters
Permission: read‑only, writes in isolated worktree
Verification: run relevant tests only
Stop: no new failures, insufficient evidence, budget reached, manual decision needed
Deliverables: failure classification, reproducible command, candidate PR, remaining risk

This clarifies what the loop can see, modify, how long it may run, when it must stop, and when a human takes over.

State Memory

Relying solely on conversation context is fragile; context is truncated, overwritten, and humans forget details. A loop needs an external state carrier such as plan.md, an issue, a board, or a simple markdown file that records the current goal, attempts, verification results, prohibited actions, and next steps. Example snippet:

## Current goal
Fix order‑export CI failures from the last 24 h.

## Attempts
- Read job 1324 log → timeout error.
- Checked recent 20 commits → possible batch‑query change.
- Increased timeout → failed, rolled back.

## Verified
- Unit test `OrderExportTest.test_large_batch` reproduces failure.
- Small‑batch export works.

## Prohibited
- Do not change permission model.
- Do not alter export file format.

## Next steps
- Compare SQL before/after batch query.
- If unresolved, hand to human for slow‑query analysis.

This file lets the next loop iteration pick up where the previous left off and lets humans quickly understand the state.

Human in the Loop

Loop Engineering is often misread as removing humans. In reality, stronger loops push human judgment earlier, turning it into explicit rules, templates, permissions, budgets, and stop conditions. Graham Neubig’s example shows a loop that first organises information, prioritises tasks, and then lets a human decide which tasks to delegate.

Conservative principles for adoption:

Read‑only first, write later.

Low‑risk tasks before core paths.

Low frequency before high frequency.

Manual confirmation before automatic merge.

Define stop conditions before continue conditions.

7‑Day Pilot Plan

Day 1: Choose a low‑risk scenario (e.g., CI triage, doc‑link check, fact‑checking).

Day 2: Write a task card describing input range, permissions, budget, stop conditions, and deliverables.

Day 3: Encode project rules, common pitfalls, verification commands, and output formats as Skills.

Day 4: Add state memory using plan.md, an issue, or a board.

Day 5: Run the loop manually step‑by‑step and note automation gaps.

Day 6: Add automatic trigger with fixed frequency and budget limits; route output to a human inbox.

Day 7: Retrospective – measure saved manual triage time, false‑positive rate, and whether the loop’s evidence is sufficient for a quick human review.

Metrics to track include hit rate, false‑positive rate, rollback rate, token/runtime cost, and evidence review time.

Conclusion

Loop Engineering may be a fleeting buzzword, but it signals a real shift: from “how do I phrase the next prompt?” to “how can the system reliably and continuously perform this class of tasks?” Prompts remain useful, but they become components of a larger, verifiable engineering system where human responsibility stays central.

Key Images