Xiaohongshu's Agentic Release Harness: Managing Overnight Deployments at Scale

Why Cross-Day Releases Are Hard

A single release of Xiaohongshu's core recommendation service spans more than ten deployment groups, thousands of pods, and lasts over ten hours on average. After the staged rollout, the team must wait for the next day's real traffic to accumulate enough samples to confirm recommendation quality before pushing to all groups. The individual steps—build, test, deploy, detect—are already automated. The difficulty lies in the gaps between steps: who confirms facts, checks conditions, and triggers the next step after the previous platform produces a result; who resumes the task after hours of waiting; who decides which actions become invalid when plans change.

These tasks exhibit two characteristics:

Long-Running: Validation depends on real time; the task must persist across sessions, process restarts, and execution-instance switches.

Long-Horizon: Steps are strongly dependent; the actual result of the previous step determines whether the next step is valid.

The environment is not static: platforms may time out, metrics may fluctuate, incidents may occur, and the release scope may be adjusted mid-flight. An erroneous action already applied to real traffic cannot be undone by rollback; a successful API call does not guarantee the action took effect as intended. Therefore, a reliable release system must do five things:

Always persist the task's current state.

Advance only based on the latest facts.

Enforce release scope, step order, and pass conditions so they cannot be bypassed by model output.

Verify the actual effect of every external action.

Safely recover from interruptions and unknown states.

These requirements cannot be satisfied by a single model session alone.

SKILL Transfers Knowledge But Cannot Bear Execution Responsibility

The team initially wrapped fixed actions in scripts and later compiled the complete release knowledge into a SKILL document of nearly 200,000 characters for the Agent to read during execution. As coverage expanded, new steps, conditions, and prohibitions were continuously added. The model could understand the process and call platforms, but constraints did not become reliable. One model hallucination caused a missing parameter, resulting in a version intended for canary release being pushed to all deployment groups.

This incident exposed the boundaries of a prompt-based approach:

SKILL can describe "must canary" but cannot enforce that every platform call carries the correct scope.

Context can record conversation but cannot serve as the authoritative state for a cross-day task.

The model can generate a plan but cannot guarantee the plan remains valid after environment changes.

Tools can return call results but cannot prove the actual effect in production.

The model can confirm its own judgment but cannot replace independent authorization and risk constraints.

Adding more prompts only increases the knowledge the model must understand; it does not produce persistence, concurrency control, idempotency, recovery, or effect verification. Release knowledge belongs in SKILL; execution responsibility must be delegated to a Harness outside the model.

How Harness Guarantees Cross-Day Execution

3.1 Overall Architecture: Connecting Agent and Production Environment

The Harness sits between the Agent and the production environment, responsible for running and controlling long-horizon tasks. It represents a release as a persistent, computable, recoverable task and places the constraints that must hold stably into a deterministic execution path. The system comprises four core entities:

Process Template: Defines release steps, preconditions, pass conditions, time conditions, action parameter constraints, and authorization boundaries.

Task: Stores a release's parameters, progress, platform results, pending actions, and confirmed decisions.

Engine: Derives the next step from the template and the task's current state, executes actions, and writes back results.

Proposal: The Agent generates adjustment plans based on the live situation, listing changed facts, affected steps, and potential external actions.

The Agent reads the task and external evidence, processes logs, metrics, changes, and natural-language intent, and performs anomaly analysis and proposal generation. It does not directly assemble production platform calls. The Engine issues actions solely based on the template and the confirmed state in the task. The model handles information that cannot be exhaustively predefined; the deterministic path controls release scope, step sequence, and pass conditions.

3.2 Task: Persisting Context Beyond Sessions

For cross-day releases, a session can serve as an interaction entry point but not as the state source. The target, parameters, step progress, platform results, pending actions, and confirmed decisions are written into an independent task, making it the authoritative state of the process. When a task is initiated, the process template version and parameters are fixed as the execution baseline; subsequent template changes do not affect the running task. After each step completes, the new state is persisted immediately. When a session ends, a process crashes, or the execution instance switches, the successor simply reads the current task and continues from the last confirmed position.

Process advancement, platform callbacks, and ad-hoc adjustments may write to the task concurrently. Multi-writer updates are coordinated via optimistic locking with version numbers (CAS). On conflict, the executor reads the latest version and re-evaluates; stale results cannot overwrite newer progress. The task stores facts that have already occurred, not an Agent's memory of the history. The Engine, Agent, progress dashboard, and release report all work from the same single source of truth.

3.3 Template and Engine: Continuous Advancement Based on Current State

The process template declaratively defines steps, dependencies, pass conditions, time conditions, and action constraints. The Engine advances tasks in a manner similar to a Kubernetes Reconciliation Loop. On each scheduling tick, it reads the task's current state, compares it with the template's desired state, and computes which steps are executable now. If conditions are met, it registers the action; if not, it preserves the context and records the next check time.

Scheduled starts, staged rollouts, and next-day effect observation are all expressed through time conditions. During waiting periods, no session or thread needs to be held. When the check time arrives, the Engine re-reads the task and platform state and decides whether to proceed. Consequently, interruption recovery does not rely on historical replay. After a process restart or instance replacement, the Engine recalculates from current facts: already confirmed steps are not re-executed, and unsatisfied conditions are not skipped.

3.4 External Actions: Grounding in Actual Effects

Merely persisting task state is insufficient to guarantee reliable external actions. The release platform may have created a release order while the execution process exits before writing back the result. Restarting and re-calling could cause duplicate releases; assuming success could skip an unfinished action. Before emitting an external action, the Engine first writes the action and a stable identifier into the task via a Transactional Outbox. The execution instance then claims the identifier and calls the platform. After an interruption, the successor reuses the same identifier; if the result is unclear, the action remains unconfirmed and the Engine queries the platform for the actual effect.

The system adopts at-least-once delivery plus effect verification, not exactly-once. Stable identifiers, platform idempotency, and fact queries together handle duplicate calls and unknown results. Subsequent steps rely only on platform facts already written back to the task. A successful API response does not directly become the basis for continuing the rollout.

3.5 Proposals: Letting the Agent Handle Changes Within Constraints

The fixed main path can be driven by the template and Engine, but exceptions and ad-hoc adjustments during a release cannot be fully enumerated in advance. Examples include determining whether a pipeline failure is related to the current change, whether a metric anomaly warrants continued observation, or how "only deploy a subset of groups tonight" should affect a running task. All require situational understanding.

The Agent reads task progress, platform results, related changes, and detection conclusions to form analyses and candidate resolutions. For operations that must modify the task, the system uses a two-phase "plan and apply" approach: the Agent first generates a Proposal listing the facts to be modified, affected steps, and potential external actions. After the Proposal is confirmed, it is written into the task, and the Engine recalculates subsequent steps based on the new state. If the task state has changed during confirmation, the original Proposal is not applied directly; instead, it is re-simulated against the latest scene. After application, invalidated old actions are discarded, and incomplete steps are rescheduled.

Whether a judgment can be made autonomously by the Agent depends on four admission criteria:

Observable: Inputs, platform effects, and anomalies have queryable evidence.

Stoppable or Rollbackable: Explicit stop or rollback actions exist with acceptable latency.

Controllable Impact Radius: The action scope has a definite upper bound and is constrained by the task baseline.

Reliable Judgment: Validated through offline evaluation, shadow runs, or online result verification.

Only when all four criteria are met and explicit authorization is granted does the judgment enter the autonomous execution scope. Otherwise, the Engine halts at the position with minimal impact, the Agent assembles evidence and candidate plans, and the responsible human makes the decision.

How a Release Spans Day and Night

Daytime: Changes and quality evidence enter the task. The Agent tracks MR review status, performs automated reviews, aggregates compilation, packaging, self-test, and effect-diff validation results, and assists in analyzing pipeline failures. The task continuously records which conditions are satisfied and which are still missing.

At the scheduled time, the system freezes the execution baseline. The Engine determines the artifact from the mainline version that passed build and quality validation, fixes the template and parameters, checks release restrictions, and starts the task. Release scope and step constraints come from the task baseline.

Nighttime: The Engine continuously advances. Deployment groups roll out and are verified sequentially according to the template. Platform results are written back to the task; when observation conditions are not yet met, the Engine records the next check time and ends the current execution. When the time arrives, a new execution instance continues from the task's current state.

On exceptions, the system confirms facts first. When the platform times out, the Engine queries whether the action took effect; when metrics fail, subsequent steps pause, the Agent gathers related changes, metrics, and logs, and generates candidate resolutions. Process adjustments enter the task via Proposals.

Next day: Effect validation decides whether to push to all. After the recommendation effect completes new-vs-old comparison, the Engine judges whether to continue based on platform results and template conditions. Once all deployments finish, the task's execution trajectory, verification results, exceptions, and decisions are summarized into a release report.

Code repositories, build systems, quality platforms, release platforms, detection platforms, and IM still perform their respective duties. The Harness connects these systems so that every result becomes the input for the next step, every wait has a resumption point, and every action has a verifiable effect.

Expanding Autonomous Scope from Execution Results

In current production observation, each release still requires about two manual interventions on average. The count is not the only metric; the reasons matter: missing facts, insufficient tool capabilities, incomplete process constraints, unverified model judgments, or decisions involving business risk trade-offs. The team addresses these through two loops:

Execution loop (current task): Observe state, advance steps, check results, and recover after anomalies or interruptions.

Governance loop (future tasks): Record the cause, judgment basis, handling action, and final outcome of each issue to fix templates, add tools, and build model evaluations.

Process gaps and configuration issues feed into templates and rules; missing observability drives platforms to add fact interfaces; recurring anomalies become evaluation samples. Judgments suitable for the Agent first undergo offline evaluation and shadow runs, then online result verification. Only after satisfying admission criteria and obtaining authorization are they incorporated into the autonomous execution scope of subsequent tasks. The degree of agency comes not from design but from actual verification.

Reflections and Outlook: Toward Autonomous Delivery

This release practice yielded three insights:

Long-horizon execution: Task state must exist independently of sessions. The Harness holds progress, waits, and pending actions, enabling the process to continue across time and execution instances.

Decision-execution separation: The deterministic path is executed by the Engine; situational understanding and anomaly analysis are done by the Agent; when the process must change, Proposals and confirmations write into the task.

Continuous evolution: Runtime results are used not only to verify the current execution but also to examine whether templates, evaluation criteria, and authorization boundaries are reasonable.

The current system can continuously advance within established processes and authorization boundaries; decisions beyond those boundaries enter controlled confirmation. In the next phase, recurring and verifiable judgments will be converted into system capabilities through the governance loop. Runtime evidence will in turn refine goals, stop conditions, evaluation criteria, and risk boundaries.

Along this direction, the release system will evolve in three dimensions:

Horizontal replication: Strengthen generic execution capabilities and business process configuration to extend to more businesses.

Vertical integration: Connect upstream to requirements and code generation, downstream to online inspection and attribution, linking the entire delivery process from change intent to online result.

Autonomous evolution: Gradually expand the scope of verified and explicitly authorized autonomous decisions.

Autonomous delivery expands the range of decisions the system can independently complete based on evidence. Goals, risk boundaries, and authorization remain defined by explicit responsible parties.

Conclusion

Large language models enable systems to understand vague intent, handle unstructured evidence, and form plans in situations that cannot be exhaustively enumerated. But for these capabilities to enter production, they must pass through a reliable execution mechanism. What the Harness guarantees is not that the model is always correct, but that when the model errs, the process is interrupted, or the environment changes, the task stops at a position where it can be judged and recovered, and continues execution based on real results. This is the key step for Agentic release to move from "able to call tools" to "able to reliably complete tasks."