How Multica Agents Acquire, Execute, and Report Tasks: Inside the Task Lifecycle

Problem to Solve

An agent dispatched a task goes from "waiting to receive" to "finished reporting". Although the flow seems simple, three hard engineering problems arise: distinguishing normal long tasks from silent deadlocks, automatically categorising diverse failure reasons (key expiry, quota exhaustion, network loss, missing model, etc.) to decide which failures merit retries, and showing real‑time progress on the frontend without flooding the network.

Mental Model: Task as a State Machine

In Multica, an issue can be assigned repeatedly, each assignment creating a new task. The task state machine includes the events task:queued, task:dispatch, task:running, task:waiting_local_directory, task:completed, task:failed, and task:cancelled. These events are broadcast from server/pkg/protocol/events.go and directly drive how the server recovers stuck tasks.

Deep Dive 1 (Free): Three‑Level Timeout to Distinguish "Slow" from "Dead"

Fixed wall‑clock limits (e.g., 1 hour) are unsuitable because AI tasks vary widely in duration. Multica uses a hierarchical watchdog: the outer scheduler scans every 30 seconds (via TickInterval) and applies two coarse thresholds—5 minutes for a dispatched task that never starts, and 2.5 hours for a running task that exceeds expected time.

Beyond this, two daemon‑side local watchdogs handle finer cases:

// MULTICA_AGENT_IDLE_WATCHDOG defaults to 30 minutes
// Terminates only when the agent is silently dead (no events produced)
// Failure reason: idle_watchdog

// MULTICA_AGENT_TOOL_WATCHDOG defaults to 2 hours
// Terminates when a tool call produces no output for a long time (suspected dead subprocess)
// Failure reason: tool_watchdog

The key insight is that as long as the agent continues to emit events (messages or tool calls), the daemon will not consider it timed‑out, regardless of wall‑clock duration. The global timeout MULTICA_AGENT_TIMEOUT defaults to 0 (no limit), meaning activity equals liveness.

Anti‑intuitive: an agent that runs for 90 minutes but keeps sending messages is never killed, whereas a silent 30‑minute agent is terminated by the idle watchdog. Multica judges activity , not raw duration , shifting from wall‑clock timeouts to heartbeat‑style liveness checks.

Deep Dive 2: Automatic Classification of 21 Failure Reasons

After a task fails, Multica must decide whether to retry automatically. Because 13 adapters return heterogeneous error formats, a pragmatic string‑based classifier maps any free‑form error message to one of 21 predefined reasons (e.g., ReasonTimeout, ReasonRuntimeOffline, ReasonAgentContextOverflow, etc.). Matching is case‑insensitive substring search, with more specific rules placed before generic ones to avoid mis‑classification (e.g., context_overflow must precede provider_quota_limit).

// server/pkg/taskfailure/failure.go
// Classify maps a free‑form error string to one of the agent_error.* sub‑reasons.
// Always returns a valid Reason.

Only three reasons trigger automatic re‑queue and retry: runtime_offline (daemon lost after dispatch), runtime_recovery (daemon crash‑restart recovers unfinished tasks), and timeout. All other reasons (e.g., quota exhausted, key errors) are marked as permanent failures for human handling. The classifier runs on the in‑flight write path, with regexes compiled once at init for performance.

Deep Dive 3: Why task:completed Deliberately Does Not Refresh Issue Cache

The frontend receives WebSocket events. The intuitive approach would invalidate the entire issue list on any task:* event. However, use-realtime-sync.ts contains an explicit comment that task:completed and task:failed are excluded from the generic invalidate path. Instead, they refresh the agent-task-snapshot cache (used by the agent panel) and trigger chat‑specific handlers.

Two event types are deliberately routed differently: task:message – emitted for every streamed message; it bypasses the generic invalidate to avoid flooding the network with hundreds of cache refreshes during long runs. task:completed / task:failed – refresh only the snapshot cache and chat handlers, not the whole issue list.

This precise routing follows the "invalidate, never write" principle: only the necessary cache segment is refreshed per event, preventing performance disasters that would arise from indiscriminate full‑cache invalidation. The explicit specificEvents list enumerates which events trigger which cache updates.

Conclusion

Multica implements a complete task‑lifecycle system: a state machine defines task progression, a three‑level watchdog distinguishes dead agents from long‑running ones, a string‑based classifier attributes failures to 21 reasons and decides retry eligibility, and fine‑grained cache routing delivers real‑time progress without overwhelming the network. This architecture enables agents to be dispatched confidently and monitored effectively.