Why Handoff Fails After the First Turn: Persisting Agent Responsibility Across Conversations

The Core Problem: Handoff Succeeds Once, Then Forgets

A user resets their password, gets handed from a triage agent to technical support, then returns minutes later with "still not working, 403." If the system treats this as a new turn and routes back to the triage agent, the user must repeat everything. The first handoff worked — the break happens between turns when the system loses track of who owns the conversation.

Delegation vs. True Handoff

OpenAI Agents SDK distinguishes two patterns: agents as tools: Main agent calls a specialist, gets a result, and continues owning the conversation.

Handoff : Responsibility transfers; subsequent user messages go to the new agent.

The user experience difference: "expert reports back once" vs. "from now on, you handle the user's follow-ups."

Framework Approaches to Persisting Ownership

Different frameworks store the "current owner" in different places:

OpenAI Agents SDK : Returns the last active agent in the run result; the application must persist it.

LangChain : Updates active_agent in state; cross-turn persistence relies on a checkpointer.

Microsoft Agent Framework : Uses checkpoints, stable participant IDs, and context synchronization to restore the handoff scene.

If ownership lives only in the model's prompt context, a single compression or restart can erase it.

What State to Record (And How Much)

State granularity should grow with failure cost:

General inquiry : Session ID, current owner, context reference.

Money, permissions, SLA : Add handoff stages — initiated, accepted, effective, failed.

Human takeover : Record takeover person, time, reason, and resume entry point.

State granularity should grow with failure cost.

Transferring Context Without Leaking Permissions

The author prefers a two-layer handoff package:

A concise brief: transfer reason, confirmed facts, current progress.

Full raw history and tool results retained for reference, not passed wholesale.

This mirrors the Hierarchical pattern of separating summary from evidence, but here the conversation is unfinished — the receiver must know what the user already tried (e.g., 403 after password reset) to avoid repeating steps. Critically, context ≠ permissions : seeing an order number doesn't grant refund rights; the new agent's tool access is re-validated against its identity.

Swarm: Removing the Supervisor Doesn't Remove the Need for Routing & State

OpenAI's Swarm (an experimental/teaching repo at https://github.com/openai/swarm) lets the current agent pick the next one. But run() returns messages, agent, and context — it does not save state between calls. The application must still decide:

Which agents may hand off to which.

How to persist the current owner across turns.

Timeout/loop fallbacks.

Audit trail of responsibility changes.

Swarm changes who decides in the conversation; routing, state, and failure handling still need a home.

Four Architectures Compared

Workflow : Next step decided by pre-defined code flow; reply/delivery by flow-specified exit; persists nodes, checkpoints, replay position.

Supervisor : Next step decided by central agent; reply/delivery by central agent; persists dispatch rationale, results, stop conditions.

Hierarchical : Next step decided by current level's control node; reply/delivery by current level's owner; persists cross-level constraints, evidence, escalation links.

Swarm / Handoff : Next step decided by current agent choosing receiver at runtime; reply/delivery by current receiving agent; persists current owner, handoff brief, routing log.

Real systems often mix them: Workflow for stable paths, Supervisor for triage, Handoff for specialist conversations, Hierarchical when tasks grow. The architectures can blend — responsibility cannot .

References

OpenAI, Orchestration and handoffs (

https://developers.openai.com/api/docs/guides/agents/orchestration

)

OpenAI Agents SDK, Agent orchestration ( https://openai.github.io/openai-agents-python/multi_agent/)

OpenAI Agents SDK, Handoffs ( https://openai.github.io/openai-agents-python/handoffs/)

Ilan Bigio, Orchestrating Agents: Routines and Handoffs (

https://developers.openai.com/cookbook/examples/orchestrating_agents

)

OpenAI, Results and state ( https://developers.openai.com/api/docs/guides/agents/results)

LangChain, Handoffs (

https://docs.langchain.com/oss/python/langchain/multi-agent/handoffs

)

Microsoft Agent Framework, Handoff orchestration pattern (

https://learn.microsoft.com/en-us/agent-framework/workflows/orchestrations/handoff

)

Erik Schluntz & Barry Zhang, Building Effective Agents (Anthropic) (

https://www.anthropic.com/engineering/building-effective-agents

)

OpenAI Swarm, GitHub repository ( https://github.com/openai/swarm)