ForgeX 2.0: Real AI Agents Achieve Verifiable Delivery Loop with Failure Recovery

ForgeX 2.0 is an AI software delivery control plane that manages process state, task allocation, and evidence verification. This article walks through a complete delivery loop for a visitor appointment feature, using real Codex model calls across requirements, design, development, independent test planning, approval, deployment, bug fixing, and recovery — all running in isolated macOS processes.

Six-Stage Delivery Mainline

The delivery flow consists of six stages: requirements, design, development, testing, release, and acceptance. Human approval and automatic execution are kept separate; failures can return to the corresponding stage. A diagram (Figure 1) illustrates this mainline with feedback loops.

Verifiable Appointment Requirement

The sample business goal is concrete: visitors submit name and appointment time, receive a queryable appointment ID; duplicate submissions must be idempotent, conflicting content must be explicitly rejected. This small scope covers input validation, persistence, query, and retry — sufficient to verify a real delivery.

Approval Bound to Content Version

Approvals are bound to phase, round, and a digest of the reviewed content. If requirements, design, or implementation change, downstream approvals become invalid. History remains but cannot approve new content. Front-end confirmation also checks record version; polling detects changes and clears old opinions, forcing re-read. The same constraint applies to release: development, production ops, and requirement owner must jointly approve the same release candidate. Server-side validates approver identities; one person holding multiple roles cannot alone approve. After test acceptance, the flow still requires release approval and online joint acceptance before closing the requirement.

Executor Boundaries and Leases

Executors receive tasks with a lease, isolation token, and input digest. Receipts must match the current task. After lease expiry, a late executor cannot advance the flow with stale results. ForgeX 2.0 blocks in this case, re-checks execution records, then decides recovery.

Real AI Agent Permissions Smaller Than Executor

Each phase (requirements, design, development, test planning) uses an independent model session, receiving only inputs allowed for that phase and returning structurally validated output. Requirements and design phases may only produce plans; any code modifications are rejected. Development phase may modify allowed business files; Git commit, self-test, and artifact generation are performed by a trusted executor. Login failures, call interruptions, or non-conforming responses cause task failure or block — no fixed-template fallback to success.

The current user mode reuses the local device's existing Codex login; the control plane never receives login credentials. Model-side tools are limited to a controlled workspace; generic shell, browser, and other MCP/plugin entry points are disabled. This distinction is critical: executors run fixed checks, models cannot expand their own tool permissions.

Generated appointment code runs within macOS system sandbox and Node permission constraints; HTTP service, persistence, health checks, and deployment are provided by a trusted runtime. Boundary tests actually checked out-of-bounds file access, network connections, subprocesses, and infinite loops. This isolation uses tool and process permissions under a single system user; it covers the current sample's verified boundaries and does not claim arbitrary malicious code safety certification.

Test and Deployment Handoff Same Artifact

Development executor completes Git commit and actual self-test, then uploads raw artifact bytes, commit hash, and SHA-256 digest. Control plane confirms storage before accepting the development completion receipt. Independent test and deployment executors download the referenced artifact via HTTP and verify commit and digest — no shared working directory.

Current persistence acceptance uses four independent Node processes: requirements/design/development, independent test, test environment deployment, release/recovery. Each worker has a private directory and signed identity. All four run on the same macOS device, proving process separation and HTTP artifact handoff, not cross-device deployment acceptance.

Test agent responsibility is clearly limited: map accepted acceptance criteria to supported test probes. Real creation, invalid input, idempotency, query, and fix verification are executed by independent trusted executors via business HTTP requests. Development executor cannot submit its own independent test conclusion.

Execution results are signed with local Ed25519 private key; signature covers task, input digest, isolation token, and actual output. Signature confirms result origin and integrity; business correctness is still judged by actual probes. Current probe protocol targets the appointment sample; extending to other businesses requires adding corresponding verification capabilities.

Network Loss Recovery Restores Original Result

Returning to the opening failure: release occurred but confirmation was lost. Restarting and re-executing from scratch could create a second release; directly marking success could mask a truly incomplete operation.

ForgeX 2.0 adds SQLite persistence for the local full delivery flow, saving requirements, approvals, objections, tasks, notifications, bugs, and artifacts. Control plane restart resumes the original flow. Repository also retains a PostgreSQL store, but this independent-process recovery acceptance uses local SQLite.

Workers maintain local execution logs: first save "acquired, executing", after execution save result and signed receipt, then request control plane confirmation. Only on confirmation success is the pending log cleared.

Thus, when confirmation is lost there is a deterministic recovery path: resubmit the already-saved identical signed receipt. When development artifact upload confirmation is lost, the system verifies and re-uploads the already self-tested original artifact — it does not re-invoke the development agent to generate another code version.

The real acceptance on 2026-10-12 injected control plane forced stop, upload confirmation loss, release confirmation loss, network partition, and release worker forced stop. After recovery, two confirmation requests carried identical signed receipts, resulting in only one release identity; deployment process, environment identity, commit hash, and artifact digest remained consistent.

2 receipt submissions, 1 actual release. This result corresponds only to the injected and verified failure scenarios.

If a process exits before receipt persistence, the system attempts to verify trusted deployment records, environment, and health status. When external result cannot be confirmed, the flow stays blocked for operator verification; production-simulated release retry still requires fresh three-party approval. Unknown state has an explicit exit — no guessing.

Rework and Bug Fixes in Same Delivery Chain

When online acceptance is rejected, ForgeX enters recovery approval. After three-party approval, a controlled executor restores the previously accepted artifact, verifies health status and digest, then enters development fix.

After requirement closure, users can submit linked bugs; system creates a new fix requirement. Fix goes through requirements, design, development, test, and approval again; original requirement and its approval evidence are retained. Prematurely marking a bug as resolved is rejected.

The 2026-10-10 real AI acceptance covered this round-trip: initial delivery, post-closure bug submission, fix release, online acceptance rejection, old artifact restoration, re-development, and final acceptance. The fix added appointment confirmation notes effective in create, retry, and query responses; old appointments remain queryable.

The entire exercise used 11 distinct Codex sessions, leaving two closed requirements, one resolved bug, and 33 historical approval records. These approvals were submitted by six different test roles via API to verify permissions and flow, not representing real business sign-off.

Process Evidence Back to Workbench

Backend constraints must be visible on the page. 2.0 converges navigation to Workbench, Delivery Requirements, and My Approvals; details show six stages (requirements, design, development, test, release, acceptance) and directly indicate which role is currently waiting and what the next step is.

Design review, rework, and recovery retain dedicated stages. Review area shows target requirement, material version, and current identity; structured plan allows reading diffs between versions. Executor pages display heartbeat, current task, and login verification; without login evidence it shows "Not yet verified" to avoid mistaking device online for model execution readiness. Figure 2 shows the independent design review page from browser fixed-sample acceptance; real model execution has separate acceptance records.

Current Completion Scope and Re-verification Entry Points

As of 2026-10-12, 2.0 has completed the local isolated delivery closed loop described here, leaving real model, fault injection, and browser regression evidence. Verification results are categorized as follows:

Regular full regression: 770 passed, 0 failed, 9 conditionally skipped

Type checking and front/back-end build: passed

Browser end-to-end regression: 6 passed, agent used fixed sample responses

Real AI full round-trip: 11 independent sessions, covering design re-review, bug fix, and recovery

Real Codex persistence acceptance: 4 model sessions; restart preserves flow, confirmation retransmission without duplicate release

The 9 conditional skips are 5 real PostgreSQL scenarios, 3 Windows-only scenarios, and 1 explicit opt-in real-time model scenario. Real model verification uses dedicated commands; cannot be substituted by regular regression numbers.

This implementation currently supports only Codex and the macOS appointment sample; Claude, real customer production environments, and real CI/CD are not yet integrated. Repository remains tagged 0.1.0 pre-release; the article's 2.0 refers to the development branch and this round of delivery capabilities. Arbitrary tech stacks, cross-device scaled operation, and production deployment each require their own integration and acceptance.

Prerequisites: Node.js 24, locally supported Codex CLI, existing login. Re-verification commands:

npm run p0:ai:sample # Real AI full round-trip
npm run p0:durable:verify # Independent process and fault recovery
npm run p0:durable:console # Persistence control plane entry

Beyond the console, four workers and web must be started per the repository's durable worker guide. In manual use, each gate is reviewed by the corresponding role; test roles in automated acceptance are only for isolation rehearsal.

ForgeX 2.0 achieves letting real model-generated artifacts gradually enter a verifiable delivery flow. For new business integration, the most valuable work is concrete: define acceptance criteria, build trusted test and deployment adapters, then run through failure and recovery scenarios one by one.

Implementation and Acceptance References

Version baseline: 2.0 branch 49a6383 · data verification date 2026-10-12

Process and constraints: docs/product/2.0-P0.md; docs/product/2.0-AI-Agent.md

Real model and bug fix: docs/verification/2.0-AI-Agent.md

Persistence and fault recovery: docs/verification/2.0-durable-codex.md

Latest regression and pages: docs/verification/2.0-delivery-interface.md

Startup and usage: docs/deployment/2.0-Durable-Codex.md

Main implementation: lifecycle-workflow.ts; lifecycle-runtime.ts; lifecycle-codex-adapter.ts; sqlite-lifecycle-repository.ts