Backstage 1.55 Task Recovery: Why Restored Jobs May Still Repeat External Operations
Backstage 1.55 adds task recovery for Scaffolder, but recovery only resumes from saved step state and workspace snapshots; external operations can still repeat if they succeed before checkpoint persistence, requiring action authors to implement idempotency and resource verification.
Backstage 1.55.0 (released 2026-09-15) promotes Scaffolder task recovery to a formal configuration. When a workflow process dies after a repository creation request succeeds but before the result is recorded, the recovery logic must decide whether to continue to the next step or re-create the repository. This article dissects the mechanism and its limits.
Recovery targets lost-heartbeat tasks, not all failures
Scaffolder executes software templates. A step is a flow step; an action is the concrete implementation (generate files, create repo, register service). Task state and step state are distinct. A task in processing has been claimed by a worker. With scaffolder.taskRecovery.enabled, if the task's heartbeat expires, recovery logic can move it back to open for re-claim. The timeout is a heartbeat-liveness threshold, not a per-action execution deadline.
Source entry points in v1.55.0:
plugins/scaffolder-backend/src/scaffolder/tasks/TaskWorker.ts— runTaskRecoveryLoop() scans for recovery, runTaskClaimLoop() claims tasks. The former calls recoverTasks() via the task proxy; the latter calls runOneTask() to hand off to the executor. Thus detection, re-claim, and actual restart are sequential; the recovery threshold cannot be treated as a business recovery-time SLA.
Re-claim does not restart blindly. With recovery enabled, the executor reads persisted step state, skips completed steps, and restores their outputs so later steps still receive prior results. This mechanism addresses mid-run interruption. A task that already threw a business error and entered a failed terminal state is not the same as a processing task that lost its heartbeat. Enabling recovery does not mean all failed tasks retry infinitely, nor does it auto-fix bad parameters.
Action return ≠ step completion
Recovery needs two artifacts: step outputs (e.g., repo URL) and workspace (generated code, config, docs). The URL surviving does not bring files back. Backstage handles them separately: task recovery does not persist workspace files by default. To retain files across restarts, you must install, register, and configure a workspace provider. Merely naming a provider without wiring the module yields no file recovery.
In NunjucksWorkflowRunner.ts, executeStep() calls serializeWorkspace() after the action returns, then updateStepState() saves completed and outputs when recovery is enabled. execute() reads back the workspace and skips completed steps. This ordering shifts the engine's definition of "complete" to after external API success: the workspace save precedes the step-completion record to avoid marking a step complete before its artifacts are durable. The trade-off: if workspace save fails, the step cannot be marked complete; if the worker handles the exception and marks the task failed, it falls outside the lost-heartbeat auto-recovery. If the process dies before the completion record is persisted, the task stays processing and will be re-claimed via heartbeat expiry — at which point the external operation may have already succeeded, causing a repeat.
Workspace storage is therefore an execution dependency, not an optional backup. The built-in database provider has a 50 MB limit, suited for dev; production should use GCP or similar external storage. Platform teams must verify that snapshots write and read reliably, and that actions can safely re-run — not just that a task row exists in the DB.
For templates that depend on prior generated files, acceptance testing can deliberately deny the new worker the original process's local directory. Missing files only indicate the workspace recovery path is broken: provider not enabled (config gap), enabled but snapshot failed (write issue), or snapshot exists but unreadable (read/storage availability). Task-record recovery alone cannot distinguish these.
Checkpoints save known results; external resources still need verification
A step may contain multiple external operations. The whole step being incomplete does not mean every operation must re-run. Custom actions can use ctx.checkpoint to record a segment's result: on recovery, if a saved checkpoint value exists, it is reused and the callback is skipped.
However, checkpoints do not make the external service and Backstage DB a single transaction. In v1.55.0 source, when no reusable result exists, the order is: run callback, then persist the success value to the checkpoint. Hence a window exists between external success and checkpoint persistence where the result is unrecorded .
The article illustrates with a custom action that creates a repo. The action wraps the create call in ctx.checkpoint with key create-repo-v1, returning { repoId: 12345 }. If the checkpoint is saved but the step incomplete, recovery retrieves repoId and skips the create callback. If the repo was created but the process exited before the callback returned, the checkpoint lacks the result and recovery may invoke create again.
The example action also persists the request and expected attributes in a platform business DB, then writes the binding between request and repo ID after the create API returns 12345, finally returning the ID from the checkpoint callback. If the binding is persisted but the checkpoint not yet saved, recovery reads the business binding, queries the repo by ID, matches org/name/visibility, and returns the same repoId — avoiding duplicate creation.
But the binding itself has an unpersisted window. If create succeeds but the binding is not saved, the request record only proves intent, not that the same-named repo belongs to this run. The action then halts auto-push and reports an ownership conflict for manual resolution based on verifiable creation records; insufficient evidence leaves it pending — no auto-claim by name alone.
Query-then-create also has a race window. If the external service supports idempotency keys, use them per the interface contract; if only unique names are available, handle concurrent creation conflicts and re-read/verify the actual object. These are business judgments the action author must supply — not granted automatically by the taskRecovery switch.
Recovery also preserves the original task secrets until terminal state, but does not extend token TTL. If a token expires at query time, the action should fail explicitly, not fall back to a higher-privilege shared identity.
Portal completion must map to actual resources
Another change in the same version: github:repo:create dropped inputs it previously accepted but ignored, including default branch and branch protection. Those capabilities must be checked for ownership by later actions like github:repo:push. This exposes a concrete issue: a field in the template does not prove the external resource acquired the attribute. Old inputs being accepted is not evidence of past effectiveness. The completed state saved by recovery only means the action ran to its implementation's end.
For this template, even after skipping the create callback, you must still verify repo attributes and branch protection; completed helps the engine pick the next step but cannot replace external resource acceptance.
Test in your own environment: enable task recovery and configure workspace storage. In the custom create action, set an async wait point after the business binding is saved but before the checkpoint callback returns. Confirm the process stops there, then force-kill the worker so the processing task awaits heartbeat expiry and recovers. Do not substitute an active exception throw — that tests the normal failed path.
Observe whether recovery resumes the same repo 12345, whether completed-step outputs are usable, whether files are intact, and whether promised branch protection actually exists. Then move the interruption point earlier: after repo create succeeds but before business binding is written — expect an ownership conflict halt, no auto-push. Finally, interrupt after checkpoint save but before step completion — verify the create callback is skipped.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
