Event-Sourced Reliable Execution Framework for Multi-Agent Video Generation

Introduction

The article presents a Java reliable execution framework designed for end-to-end video generation pipelines. The pipeline involves dozens of external model calls (scriptwriting, shot generation, style reference images, audio, compositing, review), each an asynchronous "submit-poll" task taking minutes and costing real money. The framework does not touch video business logic; it ensures the orchestration process itself is trustworthy.

Five Core Problems

Because the chain is long and each step is not 100% reliable, the probability of a full successful run drops sharply. Every failure incurs real cost (re-payment, human intervention stalls). The framework addresses five problems:

Process crash: resume without duplicate charges or lost accounting.

Concurrent instances: prevent mutual interference (split-brain).

Auditability: reconstruct why a decision was made.

Compliance & budget: enforce hard limits that cannot be bypassed.

Human approval: avoid blocking the pipeline; handle timeouts deterministically.

Comparison with Existing Solutions

The author compares four categories of existing systems across seven dimensions (fact source, decision origin, cost gate, compliance gate, multi-level review, human intervention, crash recovery):

Persistent execution (Temporal, Restate, DBOS) : event log fact source, code-as-workflow replay, no built-in cost/compliance/review/human support, crash recovery via replay.

Cloud workflows (Step Functions, Durable Functions) : state snapshot fact source, DSL/code decisions, no built-in cost/compliance/review/human, callback-based human wait.

Agent orchestration (LangGraph, AutoGen, CrewAI) : checkpoint fact source, LLM-driven decisions, no built-in cost/compliance/review, interrupt for human wait, partial crash recovery.

Ad-hoc while-loop + try/catch : in-memory fact source, free LLM decisions, none of the hard requirements.

The proposed framework uses an append-only event log as fact source, rule-table-first decisions with LLM fallback, and includes all seven capabilities.

Key Difference from Temporal

Temporal relies on deterministic code replay; placing an LLM call in the workflow breaks determinism. The framework moves control-plane decisions out of the replay path: a fixed-priority rule table (R1–R11) computes actions from the projected state; only when no rule matches does an LLM participate, and its output must pass schema validation. This preserves replay determinism and avoids workflow versioning issues.

Two Design Principles

Principle 1: Event Log is the Sole Fact Source

Every state change writes an event first (write-ahead). The current state is a projection derived by replaying all events; it is never persisted. This mirrors bank ledger accounting: the ledger is authoritative, the balance is derived. Crash recovery, instance takeover, and auditing all reduce to replaying the same log.

Principle 2: Deterministic Decisions via Rule Table; LLM Only for Open Questions

Control-plane decisions (dispatch, rollback, budget, compliance, state transitions) are computed by a prioritized rule table (R1–R11) that reads only the projection. LLM is invoked only for uncovered open problems, and its output is schema-validated. This ensures correctness (no non-determinism in control plane) and saves inference cost. Rule-table decisions are fully reproducible from the event log.

Derived Conclusion: Stateless Host

Because authoritative state lives in the event log, the orchestration process (Host) holds no irrecoverable state. Any lease-holding instance can replay events and take over. This eliminates task-to-machine affinity, making failover and horizontal scaling the same mechanism. However, statelessness alone does not prevent split-brain; that requires lease + fencing epoch (detailed in §5.2).

Task Lifecycle: What the Framework Does

Submit

A task request includes goal, execution plan (stage DAG), cost ceiling, time ceiling. Synchronous validation checks DAG acyclicity, stage-to-executor mapping, reviewer mapping, non-negative cost estimates. On success, a TaskCreated event is written and taskId returned immediately.

Decision Loop

A lease-holding instance loops: (1) replay events to get latest projection; (2) feed projection to rule table R1–R11, first match yields an action; (3) if no rule matches, call LLM; (4) write an INTENT event before executing the action. Five action types: dispatch a stage, rollback upstream, suspend for human, escalate terminate, static gate before high-cost generation.

Execution & Callback

Dispatched stages run in executors; they write back only three event types via a restricted channel with idempotency keys: outcome, progress, failure. The loop yields after dispatch, waking on new events.

Finalization

After business stages complete, the framework automatically adds composition and review phases. Review yields one of four verdicts: pass, soft warning, quality rework (triggers targeted rollback), compliance block (deterministic termination).

Terminal States

Task ends in completed, failed, or escalated (semi-terminal; human can resume, e.g., approve extra budget). Four recovery triggers keep tasks moving: host startup scans active tasks, human decision write wakes task, timeout scanner detects stalled approvals, periodic lease-expiry scan triggers takeover.

Six Key Mechanisms

5.1 Crash Recovery without Duplicate Charges (Reconciliation)

Problem: crash may occur after external service charged but before outcome recorded. Replay sees INTENT without OUTCOME. Solution: reconciliation — query the external provider with the idempotency key. If already executed, write OUTCOME with artifacts and actual cost; if not, write a void marker and re-decide. A stall watchdog cancels stages exceeding a progress threshold and writes a failure event for re-dispatch.

5.2 Concurrency Control (Lease + Fencing Epoch)

Each task has a lease row with a monotonically increasing epoch. On expiry or release, epoch increments; lease row never deleted. Host writes events in the same transaction as epoch validation. An instance with a stale epoch fails validation, is explicitly notified, and exits. Two critical implementation details: (a) lease expiry judged solely by database server time, not local clocks; (b) lease renewal runs on a dedicated single-threaded scheduler, never blocked by heavy jobs.

5.3 Decision Auditability

Rule-table decisions are reproducible from projection + rule table, so not logged separately. LLM decisions are recorded as DECISION events because they contain discretion. Each loop iteration emits at most one INTENT, ensuring minimal, clear state transitions.

5.4 Budget Enforcement (Pre-dispatch Cost Admission)

Before every dispatch, the framework sums spent cost, estimated inference cost, and estimated dispatch cost. If the sum exceeds the hard ceiling, the task is escalated with reason "pre-dispatch budget insufficient". This gate applies to both rule-table and LLM paths. Failures must also report actual spend so the cumulative total stays accurate.

5.5 Compliance Gates That Cannot Be Bypassed

Two triggers cause deterministic termination: pre-generation compliance gate failure, or final review verdict "compliance block". These do not enter rollback chains and consume no rework budget. Quality issues go through rollback and consume rework budget; compliance issues stop permanently. Compliance gates use a version anchor: they key off the latest artifact registration sequence, so each artifact version is checked exactly once, preventing re-check waste and stale-approval reuse.

5.6 Human Approval Without Blocking (Suspend Protocol + Timeout Scanner)

When human input is needed, the loop writes a "wait for human" INTENT, releases the lease, and exits. The task sits idle, consuming no resources. Human decisions are validated (existence, ownership, open, type) and written atomically; first decision wins. A timeout scanner periodically marks overdue suspensions as timed out, but does not decide the outcome — the decision loop, upon waking, applies the pre-configured timeout policy. Separating timeout marking from outcome decision avoids dual decision-makers.

Built-in Two-Phase Review

Phase 1: compliance and technical layers run via rule engine in parallel, no LLM. Hard failures short-circuit; verdicts defined by an explicit whitelist. Phase 2: content and business layers use models for subjective judgment. Both phases converge to four fixed verdicts: pass, soft warning, quality rework, compliance block. Fixed enum enables deterministic downstream branching.

Quality rework triggers targeted rollback governed by three rules: (1) invalidate dependent artifacts transitively; (2) roll back to the most upstream common ancestor to avoid duplicate rollbacks; (3) merge multiple issues into one rollback round, counted against a rework-round limit. Exceeding the limit escalates to termination. Compliance and quality paths remain fully separated.

Framework Constraints (Design-Level Invariants)

Deterministic control logic (rule table, rollback, verdict aggregation) is immutable; only parameters (rework limit, gate config) are tunable.

Event schema is append-only; new fields are optional, major changes require versioned converters; history never rewritten.

Graceful shutdown guarantees event durability and state recoverability, not in-flight work completion.

Projections are read-only for display; decisions must never read projections.

One INTENT per loop iteration; no batch dispatch.

Two distinct write paths: Host uses fenced transaction; executors, timeout scanner, human gateway use idempotency keys. Merging them would lose one protection model.

Mapping Back to the Five Problems

Crash recovery + no double charge + no lost accounting → event replay + reconciliation (§5.1).

No instance interference → lease + fencing (§5.2).

Decision auditability → rule-table reads projection only + LLM decision logging (§5.3).

Budget enforcement + compliance non-bypass → shared cost admission + deterministic compliance termination (§5.4, §5.5).

Non-blocking approval + timeout handling → suspend protocol + scanner only marks, loop decides (§5.6).

All five derive from the single choice: fact source = replayable event log, not current state value. Cost gates, compliance, and layered review are domain-specific layers on that foundation.

Unsuitable Scenarios & Open Work

Not for low-latency online services (replay/event overhead), pure agent chat (LangGraph territory), or generic workflows without LLM discretion, cost ceilings, or compliance reviews (Step Functions simpler). Two incomplete areas: (1) database middleware behavioral fidelity — client-side parsing contract tests incomplete; (2) cross-task resource scheduling — budget and concurrency admission are per-task; global multi-task scheduling unresolved, currently guarded by queue backpressure.

Appendix: Terminology Quick Reference

12 Event Types : TaskCreated, Intent, Decision (LLM trace), Outcome, ExecutionResult, Progress, Failure, SystemMarker, HumanDecision, Rollback, Escalation, PartialResolution.

Task States (7) : Queued, Running, WaitingHuman, Repairing, Escalated, Completed, Failed. Terminal: Completed, Failed, Escalated (semi-terminal, human can resume).

Review Verdicts (4) : Pass, SoftWarning, QualityRework, ComplianceBlock.

Human Decisions (4) : Approve, Resume, Abort, AddBudget.

Idempotency Keys : OperationKey (spans one external call, used for reconciliation, cancellation, terminal state), EventDedupKey (single logical write, unique constraint).

Recurring Terms : Lease, Fencing Epoch, Write-Ahead, Fold (replay), Projection (derived current state, not persisted).

Appendix: Problem–Mechanism Lookup Table

Crash double-charge → Intent without Outcome triggers provider reconciliation (§5.1)

Multi-instance split-brain → Lease + monotonic epoch, write-time validation in same transaction (§5.2)

Decision not reproducible → Rule table reads projection only; LLM decisions logged separately (§5.3)

Budget overrun → Pre-dispatch cost admission, shared by deterministic and LLM paths (§5.4)

Compliance bypass → Gate failure and compliance block cause deterministic termination, no rollback (§5.5)

Approval blocks pipeline → Suspend releases lease; timeout scanner only marks, loop decides (§5.6)

LLM output breaks determinism → Control plane uses rule table; LLM output must pass schema validation (§3.2)