Skip to content
Long-Running Tasks: Why Timeout Hacks Fail & What Actually Prevents Crashes

Long-Running Tasks: Why Timeout Hacks Fail & What Actually Prevents Crashes

This article explains why treating hour-long tasks as extended short requests fails, detailing required patterns: bounded waiting with heartbeats to prevent connection drops, two-level timeouts (per-step and total), partial failure semantics for batch jobs, explicit artifact ownership to avoid cross-session contamination, and acknowledges the critical lack of checkpointing/resume, advising task decomposition instead.

James' Growth Diary
James' Growth Diary
James' Growth Diary

01 | Global View: Three Ways Tasks Disappear Mid-Run

The author frames long tasks as an ultra-marathon. A table maps race concepts to engineering mechanisms:

Cut-off time → Total timeout (max overall duration)

Split timing points → Per-step timeout (detect stuck steps)

Supply stations → Intermediate checkpoints (save progress, enable resume)

Check-in punches → Heartbeats (prove liveness, not progress)

Dropping out → Explicit abandonment (clear signal, not silent failure)

Segment results → Partial results (outputs from completed segments)

Finish-line baggage → Artifact ownership (which session owns output files)

Three real failure scenes:

Mistaken for dropout: Task still running but middle layer kills connection due to inactivity. The task isn't broken; the channel was recycled.

Reached finish but artifacts lost: Task shows success but artifact directory empty or contains previous run's files. Root cause: artifact ownership confused by concurrent sessions relying on last-modified timestamps.

All good until last segment fails: 8-hour task fails at hour 7; retry means another 8 hours. No intermediate checkpoints → everything before failure wasted.

02 | Why Common Approaches Break: Three Fractures

Fracture 1: Assuming "Results Soon"

Most stacks are designed for seconds-to-minutes: timeouts in minutes, idle connections recycled, artifacts bound to session. Scaling parameters doesn't fix the model mismatch. Long tasks need bounded waiting, liveness proof, two-level timeouts, explicit artifact ownership.

Fracture 2: Binary Success/Failure Semantics

Batch job with 3 failures out of 20. If system only returns "failure", caller must re-run all 20. Correct semantics: partial failure — overall success, each item carries status, caller decides which to retry. This pushes retry decisions to the party that knows business criticality.

Fracture 3: Progress Has No Foothold

Most expensive aspect of long tasks is inability to resume after failure. Current layer lacks intermediate checkpoints and resume capability — acknowledged as a gap, not a trade-off.

Comparison of Four Approaches

Treat as one super-long call: Low implementation cost, but middle layers kill it; failure loses everything. Reject as default.

Just increase timeout: Trivial change, only delays failure. Reject as solution.

Heartbeats + bounded waiting: Prevents misclassification as dead; medium cost; keeps channel alive. Reject using heartbeats as progress signals.

Intermediate checkpoints + resume: True long-task guarantee; high cost; not yet implemented — do not pretend it exists.

03 | Core Idea: Turn One Marathon into Many Judgable Segments

Two-layer structure:

Orchestration layer: Scheduled tasks, dependencies, notifications, scheduling. Constraint: scheduling fields are a coupled group; switching modes must clear incompatible old fields.

Execution layer: Background tasks, heartbeats, two-level timeouts, artifact ownership. Constraint: waiting must be bounded; failures must pinpoint exact item.

Connection point: task identifier . No real identifier → no wait relationship. Tool failure must not block conversation — better to miss a wait than let a failed task hang the dialogue.

04 | Key Design 1: Waiting Must Be Bounded, Idle Must Self-Prove Liveness

Naive while True wait has two fatal flaws: no upper bound (waits forever on anomaly) and no data flow (middle layer assumes connection dead). Correct approach:

Bounded wait: Deadline must yield a conclusion. Conclusion can be "still running" but must be explicitly sent, not silence.

Idle heartbeats: Periodic empty signal (e.g., "N tasks still running") solely to prove connection alive.

Critical pitfall: On wait timeout, do not cancel the pending read . Cancelling may close the shared command stream, killing the live task. Instead, keep the read task and inject a heartbeat into the channel.

Illustrative code (Python & TypeScript) shows a generator yielding messages with a 300-second bound and 30-second heartbeat interval. On timeout, yields {"kind": "stillRunning"} (or still_running). Heartbeats only prove liveness — a stuck task can still emit heartbeats.

05 | Key Design 2: Two-Level Timeouts, Partial Failure

Two-Level Timeouts

Per-step timeout: Max duration for a single step. Without it, a stuck first step consumes entire budget.

Total timeout: Max duration for whole run. Without it, many steps have no upper bound.

Both required; total timeout is not sum of per-step timeouts — scheduling, waiting, retries consume time too.

Partial Failure

Batch returns overall success code (0) with per-item status ( ok or failed + reason). Caller retries only failed items. Value: retry decision delegated to business-aware caller.

Illustrative TypeScript/Python code shows syncAll iterating items, checking total deadline, calling syncOne with per-item timeout, collecting results, returning {code: 0, results: [...]}.

Artifact Ownership: The Easy Mix-Up in Long Tasks

Concurrent sessions writing files sequentially cause timestamp-based ownership to misattribute artifacts. Solution: explicit declaration by producer — who produced it, which session owns it. Cost: model must actively declare; occasional missed label only loses one artifact, but misattribution sends A's output to B.

06 | Acceptance: How a "Run All Day" Task Completes

Scenario 1 (Easy): Periodic Small Tasks

Scheduling fields changed as a group — clear incompatible old fields when switching trigger mode.

Write-then-read-back verification — config read-modify-write not atomic; must re-read to confirm persistence.

Notify humans — failures must alert, else nobody knows it stopped.

Decision point: Task dependencies must be set in pairs — half-set equals not set.

Scenario 2 (Hard): Day-Long Evolutionary Task

Confirm it runs in the right container shape — long tasks need containers that stay alive, not pooled for short chats.

Watch heartbeats, not progress — heartbeats only confirm channel alive.

Must be able to stop at safe points — no checkpoint/resume yet, so split large task into smaller independently verifiable tasks instead of one day-long run.

After finish, check artifact ownership first — verify artifacts attached to correct session before inspecting content.

Decision point: If task goes off-track halfway, abort . Without checkpoints, continuing only wastes more time.

07 | Three Layers of Change from This Design

Quantified efficiency: Bounded waits + heartbeats eliminated "misjudged as dead connection" interruptions — previously hardest to debug because they looked like random faults.

Capability push-down: Partial failure semantics delegate retry decisions to caller; platform no longer guesses business importance.

Pattern upgrade: From "increase timeout" (gambling it finishes before limit) to "segment tasks" (every segment verifiable).

Selection Guide: Four Approaches Compared

Implementation cost: Super-long call (low), increase timeout (very low), heartbeats+bounded wait (medium), checkpoints+resume (high).

Avoid misjudged disconnect: Super-long call (low), increase timeout (low), heartbeats+bounded wait (high), checkpoints+resume (high).

Loss on failure: Super-long call (all), increase timeout (all), heartbeats+bounded wait (current segment), checkpoints+resume (almost none).

Observability: Super-long call (low), increase timeout (low), heartbeats+bounded wait (medium), checkpoints+resume (high).

Currently deployable: Super-long call (high), increase timeout (high), heartbeats+bounded wait (high), checkpoints+resume (low).

Steal the "avoid misjudged disconnect" column — it solves the hardest-to-debug faults. Be honest about the last column: checkpoint/resume not implemented — don't draw it in architecture diagrams.

08 | Scenarios Not Suited & Four Pitfalls

Not Applicable

Tasks that must complete in one go, cannot pause (e.g., full reconciliation) — no checkpoints means restart from scratch; either split or find another solution.

Tasks requiring mid-run human/model swap — model degradation and switch consistency not handled.

Latency-critical tasks — long-task latency inherently unpredictable; don't use for real-time paths.

Pitfall 1: Treating Heartbeats as Progress

Temptation: Heartbeat carries active task count, looks like progress. Wrong fix: Use heartbeat frequency to judge task health — stuck tasks still emit heartbeats. Right fix: Heartbeats only for keep-alive; progress needs separate query endpoint.

Pitfall 2: Cancelling Read on Wait Timeout

Temptation: No result → cancel wait. Wrong fix: Cancel pending read — may close shared command stream, killing live task. Right fix: Keep read, inject heartbeat; real cancellation uses task's own cancel interface.

Pitfall 3: Only Total Timeout, No Per-Step Timeout

Temptation: One overall deadline feels enough. Wrong fix: Stuck first step burns entire budget, and you can't see where it's stuck. Right fix: Two rulers — per-step timeout finds "stuck at which step".

Pitfall 4: Using Timestamps for Artifact Ownership

Not just a pitfall — inevitable error in long tasks: concurrent sessions write files back-to-back, timestamps misattribute. Misattribution cost far exceeds missed label.

Action Levels

L1 (On-call): View task status, heartbeats, artifact list.

L2 (Platform team): Adjust per-step/total timeouts, add/remove scheduled tasks.

L3 (Change review): Modify wait/cancel strategies, adjust artifact ownership rules. L3 exists because changing wait/cancel behavior ripples to all long tasks.

Summary

Long tasks ≠ "increase timeout" — they need a different model for timeouts, keep-alive, artifact ownership.

Waiting must be bounded: deadline must yield conclusion, even if "still running".

Heartbeats only prove liveness, not health — stuck tasks still send heartbeats.

Timeouts in two levels: per-step finds stuck points, total controls budget; they cannot substitute each other.

Partial failure must be explicit: overall success + per-item status; retry decisions to business-aware caller.

Honest about gaps: no intermediate checkpoints, no resume, no mid-run degradation handling — so large tasks must be proactively split into independently verifiable small tasks.

True cost of a long task isn't how long it takes to finish, but how long it takes to re-run from scratch after failure.

Next article: Harness Self-Evolution: Letting Agents Improve Agents — long tasks solve "run once for a long time"; next solves "after finishing, can the next run improve itself" — relies on evaluation loop, the season finale.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

checkpointingagent platformlong-running tasksheartbeatsartifact ownershipbounded waitingpartial failuretwo-level timeouts
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.