Designing a Durable Agent Loop Architecture for Reliable AI Workflows

Loop Evolution and Durability

Matt Van Horn traces the agent‑loop evolution from ReAct through tool‑calling, orchestration loops, to loops that supervise other loops [1]. Addy Osmani decomposes a loop into automation, worktree, skill, connector and sub‑agent modules [2]. Both authors converge on durability: a loop that cannot survive a process restart is not a functional loop. Durability therefore belongs to the entire execution layer, not just the loop primitive.

Where Loops Break

Supervising other loops

Running on a schedule instead of manual triggers

Surviving process restarts, deployments and crashes

Supporting sub‑agents that may wait for hours

Providing post‑mortem observability

These are infrastructure problems, not prompt‑engineering issues. As @runes_leo notes, the most expensive part of AI programming is managing the agent loop, not writing code.

Three‑Layer Agent Loop Architecture

Loop Layer (cron + decision maker)

The loop acts like a cron that evaluates state and decides the next step. In Inngest this is expressed with createFunction and a cron schedule; each step.run() is a durable checkpoint.

export const infraHealthCheck = inngest.createFunction(
  { id: "infra-health-check" },
  { cron: "*/30 * * * *" },
  async ({ step }) => {
    const metrics = await step.run("fetch-service-metrics", async () => {
      return await fetchServiceMetrics(); // error rates, latency, memory, CPU
    });
    const assessment = await step.run("assess-health", async () => {
      return await callLLM({
        prompt: `Given these service metrics, classify overall system health as "normal", "degraded" or "critical". Explain your reasoning. Metrics: ${JSON.stringify(metrics)}`
      });
    });
    if (assessment.status === "degraded" || assessment.status === "critical") {
      await step.invoke("triage-incident", { function: incidentTriage, data: { metrics, assessment, services: assessment.affectedServices } });
    }
  }
);

Each run stores its inputs, outputs and retry count, enabling exact replay after a crash.

Skill Layer (durable workflow)

A skill is a multi‑step, retryable, composable workflow that can be deployed independently. It may call an LLM or run deterministic code.

export const incidentTriage = inngest.createFunction(
  { id: "incident-triage", retries: 3 },
  { event: "infra.incident.triage" },
  async ({ event, step }) => {
    const details = await step.run("fetch-detailed-metrics", async () => {
      return await fetchDetailedMetrics({ services: event.data.services });
    });
    const deploys = await step.run("fetch-deploy-history", async () => {
      return await fetchRecentDeploys({ since: hoursAgo(2) });
    });
    const analysis = await step.run("correlate-incident", async () => {
      return await callLLM({
        prompt: `Correlate these service metrics with recent deploys. Identify the likely root cause and severity. Metrics: ${JSON.stringify(details)} Recent deploys: ${JSON.stringify(deploys)}`
      });
    });
    await step.run("post-triage-summary", async () => {
      await slack.postMessage({ channel: "#incidents", text: formatTriageSummary({ analysis, affectedServices: event.data.services, recommendedActions: analysis.recommendations }) });
    });
    return analysis;
  }
);

The skill can be retried at the step level, avoiding duplicate LLM calls and saving tokens.

Orchestrator Layer (engine)

The orchestrator schedules crons, runs steps, manages retries, enforces concurrency limits, stores full run history, and supports hot‑deployment of new functions without interrupting in‑flight tasks. It is invisible to most users but is the foundation of durability.

Error Handling and Durability

When a step fails, Inngest retries only that step, preserving completed work. If all retries are exhausted, an onFailure hook can notify operators while the original event data remains intact for the next scheduled run.

export const incidentTriage = inngest.createFunction(
  { id: "incident-triage", retries: 3, onFailure: async ({ error, event, step }) => {
      await step.run("notify-failure", async () => {
        await slack.postMessage({ channel: "#agent-ops", text: `⚠️ Incident triage failed: ${error.message}. Will retry on next health check cycle. Affected services: ${event.data.services.join(", ")}` });
      });
    }
  },
  { event: "infra.incident.triage" },
  async ({ event, step }) => { /* same logic as skill above */ }
);

Durable checkpoints also reduce cost by preventing repeated LLM calls across retries.

Durability Requirements

Independent step retry : If step 3 of 5 fails, only step 3 is retried. A basic while True loop would re‑execute all steps.

Sub‑agent lifecycle : Parent tasks can cancel children; children may run for hours. Simple loops lack built‑in parent‑child management.

Guaranteed event delivery : Events survive agent shutdown. Plain loops lose events when the process stops.

Post‑hoc observability : Every step, decision and retry is recorded. Without checkpoints only transient logs remain.

Hot‑deploy without downtime : New function versions run alongside existing tasks. Process restarts would otherwise kill all tasks.

Concurrency control : Limit the number of concurrent instances of a skill. Basic loops have no concurrency primitives.

Comparison with Existing Tools

Some turnkey solutions provide a polished stack; others let you stitch lower‑level components together. The essential property is an architecture that remains flexible, dynamic and durable, allowing agents to evolve their own loops and skills.

Compounding Loops

Satya Nadella argues that a company’s moat is its loop, not its model [4]. Human capital (knowledge, judgment) and token capital (AI workflows, learned skills) compound together. Each improved skill feeds better signals to the LLM, freeing humans for higher‑order decisions.

Practical Guidance

When building production‑grade agent loops, first define the three layers—loop, skill, orchestrator. Use Inngest’s step‑level checkpoints, onFailure hooks, and concurrency controls to achieve durability, observability and hot‑deployment without downtime.

Agent‑Generated Skills and Review Loop

An agent can write a new skill, register it with the orchestration engine, and have it run immediately without a PR pipeline. Example workflow:

Engineer requests a nightly health‑check for a service that spikes at night.

Agent creates two functions: a health‑check loop (every 30 min) that classifies system health via LLM, and an incident‑triage skill that fetches detailed metrics, recent deploys, correlates root cause with an LLM, and posts a Slack summary. Errors such as a timed‑out metrics API trigger exponential back‑off retries; LLM failures fall back to rule‑based classification.

Agent deploys the skill; a sidecar process picks up the new function and registers it automatically.

The health‑check loop triggers the triage skill when needed; the entire flow is durable.

A weekly review function runs on Friday, fetches the past week’s run history, computes success rate and average duration, queries an LLM for performance analysis, and, if the LLM recommends changes, invokes an update‑skill function to modify the incident‑triage skill.

export const reviewSkillPerformance = inngest.createFunction(
  { id: "review-skill-performance" },
  { cron: "0 10 * * 5" }, // Every Friday at 10 am
  async ({ step }) => {
    const runs = await step.run("fetch-run-history", async () => {
      return await getInngestRuns({ functionId: "incident-triage", since: daysAgo(7) });
    });
    const analysis = await step.run("analyze-performance", async () => {
      const successRate = runs.filter(r => r.status === "completed").length / runs.length;
      const avgDuration = average(runs.map(r => r.duration));
      const incidents = await fetchIncidentOutcomes();
      return await callLLM({
        prompt: `Review this skill's performance over the past week. Success rate: ${successRate}. Avg duration: ${avgDuration} ms. Incidents correlated with real outages: ${incidents.confirmed}/${incidents.total}. False positives: ${incidents.falsePositives}. Should we adjust thresholds or classification? Provide concrete changes.`
      });
    });
    if (analysis.shouldModify) {
      await step.invoke("update-skill", { function: coreAgent, data: { prompt: `Update the incident‑triage skill with these changes: ${analysis.proposedChanges}` } });
    }
  }
);

The review loop reads orchestration history, lets an LLM suggest adjustments, and applies them via a hot‑deployed function, demonstrating continuous self‑improvement.

Developer Observability

The orchestration engine stores every run, step input, step output and retry count. Developers can inspect the dashboard to see which skill failed, the exact input that caused the error, and how many retries occurred. Each step.run() is a checkpoint and automatically observable, providing a trust layer when code is generated by an agent.

Why Durability Is Fundamental

Without durable checkpoints, a process crash forces the loop to start from the beginning, re‑issuing all LLM calls and sending duplicate notifications. This not only breaks correctness but also inflates token costs dramatically when many agents run in parallel.

References

[1] https://x.com/mvanhorn/status/2063865685558903149

[2] https://addyosmani.com/blog/loop-engineering/

[3] https://x.com/runes_leo

[4] https://x.com/satyanadella/status/2066182223213293753