Model, Runtime, Host: Untangling the Three Layers of AI Agent Architecture

Hello, I'm James. In the previous article "One Diagram to Understand Self-Developed Agent Layering" , we mapped out the seven-layer responsibility diagram. Today we won't drill deeper into a single layer; instead we'll nail down the three terms that most often muddy the diagram: Model , Runtime , and Host .

The layering diagram answers "which layers does a request traverse?" These three terms answer "within the same layer, who are you actually blaming?" Mixing them up leads to misdiagnosis during incidents, wrong capability purchases, and post-mortems that mistakenly label product accidents as model accidents.

The most common complaint on the incident scene is "the model is bad." Yet under that same phrase lie three completely different pathologies: the first character takes forever to appear — usually a cold-start connection issue; message bubbles appear out of order — typically a Host presentation-layer problem; tool calls fail — often because the Runtime never injected the configuration. In all three cases, the Model is often the most innocent party.

First, Align on a Few Terms in One Minute

The article defines a glossary of key terms:

Model : The part that only "thinks." Given context, it returns text or tells you "which tool to call next."

Runtime : The actual program running inside the container. It takes the Model's output, calls tools, modifies files, records sessions, and ensures "stop means really stop."

Host : The shell where the user lives: web page, Enterprise WeChat, command line, IDE — all count.

Tool Loop : Model says "call this tool" → Runtime executes → result fed back to Model → Model decides next step, repeating in cycles.

Cold Start : The one-time overhead of establishing a connection for the first time, reusable afterward. Not counted from the second request onward.

Turn : User sends one message, Agent finishes replying — that's one turn.

SDK : Ready-made dev package from Model vendor or Agent framework; the tool loop is usually already encapsulated.

Modal : A confirmation dialog requiring a user click to proceed. Command-line and Enterprise WeChat hosts cannot pop up modals.

Contract : The public promise of one layer to the layer above: "what I guarantee and what I don't."

(Common technical terms like BFF, SSE, ACL, Sandbox, Token, P99, SLA are used directly without explanation.)

01 | Three Complaints, Landing on Three Different Layers

An analogy: renovating a house involves three roles:

Designer (Model) : Understands requirements, produces plans, says "demolish here, install there." Does not lay bricks or run wires; has no tools.

Construction Crew (Runtime) : Takes the blueprint and actually demolishes, installs, buys materials; when you yell "stop work" they must truly stop. Does not change the plan, nor handle customer reception.

Storefront / Front Desk (Host) : Decides how customers enter (web, IM, phone), how progress is displayed, where signatures happen. Does not do construction.

This analogy directly explains the three most frequent arguments:

"Renovation too slow" isn't necessarily the Designer drawing slowly — the Construction Crew might still be en route or tools not on site. Online: first character delayed → check Runtime readiness first, don't rush to swap Models.

"Customer wants to sign in WeChat but can't" isn't the Designer's fault — the storefront lacks a signature pad. Online: Agent stuck in IM → Host lacks confirmation channel, not that Model doesn't know how to ask.

"Designer specified an outlet, but it wasn't installed" isn't a blueprint error — the Crew didn't bring the outlet that day. Online: tool call fails → Runtime didn't inject tool config into process, not that Model spoke wrong.

Remember this mapping; every subsequent section refers back to it.

When a self-developed Agent goes live, chat groups produce three "same name, different thing" complaints. All sound like "AI doesn't work," but mechanically they land on three layers.

First Ask: Which Layer Is Broken?

Wrong words / Won't think
Can think but can't act / Won't stop cleanly
Can act but can't see / Confirmation stuck
User perception broken
First ask: which layer is broken?
Model
Runtime
Host
Swap Model / Adjust Prompt
Session Reuse / Tool Loop / Cancel
Presentation Layer / Callback Channel / Entry Adaptation

Three Incident Scenes

Scene A: First character slow — Surface symptom: "Model so slow." Common misdiagnosis: swap faster Model. Mechanism truth: Runtime — every utterance cold-starts CLI connection; money spent on connection layer, not inference.

Scene B: Stuck in Enterprise WeChat — Surface symptom: "Agent won't ask human." Common misdiagnosis: Model can't use tools. Mechanism truth: Host — can't pop Modal; permission callback not degraded, human never sees the question.

Scene C: Works on Web, fails in IDE — Surface symptom: "Two sides use different Models." Common misdiagnosis: misconfigured Model. Mechanism truth: Host Strategy — MCP and auth injection paths not aligned.

Model only handles "thinking"; Runtime handles "doing in workspace and stopping cleanly"; Host handles "how humans see, confirm, and enter."

Blaming the wrong layer costs far more than a slightly weaker Model — entire weeks of debugging wasted in the wrong place.

02 | One Word Smothers Three Layers

Public products, open-source frameworks, and internal demos all love to call the entire chain "Agent." Fine for marketing, fatal for engineering — that single word smothers at least three layers.

2.1 Capability Boundaries Smudged into "Model"

When teams debate "whether to upgrade to a stronger Model," what's actually missing is often three other things: tool loop, workspace side-effects, cancellation semantics. The Model API only emits tokens; "being able to do work" happens inside the Runtime-driven tool loop, not inside the Model.

Concretely: Model reads context, says "I suggest calling 'query_order' tool with param 123." That's it — it only states intent , neither executes nor has permission to execute. The Runtime actually calls, retrieves result, feeds it back; Model then decides "done, answer" or "need another tool."

So "can do work" = Model proposes + Runtime executes + a working loop between them. Miss any one, product only chats.

Self-check for misattribution : Replace Model with a stub that always returns "I want to call tool X." If product still can't do work, bottleneck is definitely in Runtime, not Model.

2.2 The Layer That Actually Does Work Smudged into "Service"

Spinning up an HTTP process in a container that forwards prompts gets called "Runtime." It's just a thin proxy — able to forward ≠ able to execute.

A real Runtime must do at least five things:

Reuse connections keyed by session → else every utterance cold-starts, user feels "slower the more we chat."

Inject tool and credential configs into process → else Model proposes tool call but Runtime holds no key.

Drive tool loop and feed back results → else Model proposes and conversation halts mid-way.

Support cancel and preemption → else user hits stop, process keeps writing disk.

Allow audit-time identity traceback → else incident investigation finds no culprit.

Missing these, you have a "chat-only forwarder," not an "execution layer that can do work."

Self-check : Ask "After user hits stop, does the tool really stop?" If you can't answer, it's likely still just a proxy.

2.3 Entry Points Smudged into "Frontend"

Web, Enterprise WeChat, IDE, CLI, OpenAPI — all labeled "frontend" or "client." Their true engineering identity is Host : decides how events are displayed, whether confirmation channels exist, where identity comes from.

Treating Host as "pure UI" steps on two mines:

Pretending modals exist in headless channels. CLI, IM hosts can't pop confirmation dialogs; if core only knows "popup to ask," those entries deadlock.

Forking a second Agent inside IDE. Extension ships fast, but once core forks, every later change to cancellation semantics or injection rules requires dual maintenance — interest compounds quarterly.

Different Host strategies, shared core — that's normal design. The difference lies in strategy, not in core.

Self-check : Same Agent behaves differently in Web vs IDE — first suspect Host strategy misalignment, not that two different Models are connected.

With three layers stacked, "we built an Agent" becomes unusable for troubleshooting and selection.

In evaluations we steal "single core, Host strategies may differ"; reject "using product name as architecture"; reject "any AI problem → swap Model first."

Only Chat API : Steal — fast launch; Reject — workspace side-effects, true cancellation.

Self-developed full tool loop : Steal — protocol control; Reject — cost and ecosystem disconnect.

Official SDK + self-developed Host adapters : Steal — mature tool loop; Reject — pretending every Host has UI.

One Agent per entry : Steal — short-term independent iteration; Reject — month three strategy divergence explosion.

03 | Three Contracts, Each Governing a Segment

A brand name "Agent" must be split into at least Model Contract, Runtime Contract, Host Contract — without splitting, meeting minutes are just synonyms arguing.

Two Abstraction Layers: Conceptual and Implementation

Conceptual Layer (What) : What Model / Runtime / Host each guarantee. Solves: label first during discussion, no single "Agent" ruling all.

Implementation Layer (How) : Model routing, session connection pool, entry display and callbacks. Solves: ground labels on observable boundaries.

Two layers because concepts precede implementation: implementations change (swap SDK, swap IDE, swap IM), concepts stay stable, so troubleshooting vocabulary holds.

Model          Model
Runtime        Runtime
Host           Host
Web / WeCom    Web / IDE
IDE / CLI      CLI / OpenAPI
Agent          Front Gateway
Container      Container Dialog Process
SDK / CLI      Tool Loop
LLM            Inference & Generation

Model = Inference Contract; Runtime = Execution Contract; Host = Interaction & Entry Contract.

04 | Writing the Three Terms as Verifiable Clauses

Saying "we separated the three terms" is useless; they must be written as checkable clauses.

4.1 Model: Only Accept "Thinking" Part

🎯 Challenge : Business dumps latency, format, tool failures all on Model. 💡 Solution : Model clauses cover only four things — context window, tool-call protocol compatibility, content safety, billing unit price.

First-token budget contains a large chunk for connection and container pool; Model SLA shouldn't swallow the entire first-response.

Illustration (TypeScript):

// Illustration · TypeScript
// Rule to keep: Model Contract only describes the "inference" segment
type ModelContract = {
  id: string;
  maxContextTokens: number;
  supportsToolCalls: boolean;
  // Latency metric only counts inference, not connection cold-start in P99
  latencyScope: "inference_only";
};

Illustration (Python):

# Illustration · Python
@dataclass
class ModelContract:
    id: str
    max_context_tokens: int
    supports_tool_calls: bool
    latency_scope: str = "inference_only"  # Only inference, excludes connection cold-start

In plain language: Model clause acknowledges only these four items, meaning any issue involving environment, connection, or UI is not its responsibility. First-token budget has a big slice for containers and connections; Model SLA must not swallow the whole first-response — otherwise every slowdown gets "fixed" by buying a more expensive Model.

🛡 Boundary : Swapping Model won't fix "hit stop but tool keeps running" — that's Runtime's cancellation semantics. Back to renovation analogy: No matter how expensive the Designer, they won't pull the main breaker for you.

4.2 Runtime: Accept "Do" and "Stop"

Process alive ≠ Runtime ready. Container up, port open, but middle CLI connection still cold, or previous turn still hogging connection — new request still deadlocks. So Runtime clause must specify at least five conditions: session-keyed connection reuse, MCP and secret injection, tool loop driving, cancel/preemption support, audit-time identity traceback.

User's "Agent frozen" is often previous turn holding connection. So "stop cleanly" must be in contract, not just ops folklore.

Illustration (TypeScript):

// Illustration · TypeScript
// Rule to keep: Three green lights before accepting next turn
type RuntimeReady = {
  processUp: boolean;          // Process up
  connectionWarm: boolean;     // CLI / SDK connection reusable
  turnIdle: boolean;           // No old task occupying
};
function canAcceptChat(r: RuntimeReady) {
  return r.processUp && r.connectionWarm && r.turnIdle;
}

Illustration (Python):

# Illustration · Python
def can_accept_chat(process_up: bool, connection_warm: bool, turn_idle: bool) -> bool:
    return process_up and connection_warm and turn_idle

In plain language: Three lights means "process up" is only the first light. Many see container up and port open and assume ready, but the middle light "connection is warm" is still off — user still waits.

🛡 Boundary : Runtime doesn't translate data stream into IM bubbles — that's Host presentation layer. Construction Crew follows work orders, doesn't care how progress appears on storefront screen.

4.3 Host: Accept "See" and "Confirm"

Treating Host as skin and dumping core events raw to UI is the most common laziness. Host clauses must be written for two cases: with UI — use Modal and permission callbacks; without UI (CLI, Enterprise WeChat) — degrade to text options or automatic policies. Regardless of entry via browser, IM, IDE, or CLI, all must converge on the same Agent path; only strategies may differ.

Headless channels can't pop windows. So if "unified experience" is interpreted as "popups everywhere," that's designing an accident, not unifying experience.

Host is a source of truth for confirmation channels and event display — it's not skin.

🛡 Boundary : Host shouldn't rewrite a tool loop just to save effort. Core forks → interest compounds quarterly. Storefront can decide how to greet customers, but cannot hire its own construction crew.

05 | One Lookup Table to Freeze Troubleshooting Vocabulary

Translating colleagues' spoken phrases into engineering labels is the highest-value deliverable of this article.

"Model hallucinates" → Check Model first. Evidence: same prompt on isolated eval set. Don't: tweak gateway timeout first.

"First character takes forever" → Check Runtime (connection) / Container Pool . Evidence: cold-start logs, connection pool hit rate. Don't: upgrade to larger Model first.

"Tool call fails" → Check Runtime (injection) / External Capability . Evidence: did MCP config enter process? Is Token still valid? Don't: blame frontend button first.

"Bubbles out of order / truncated" → Check Host Presentation Layer . Evidence: frame protocol, byte window, queue. Don't: tune Model temperature first.

"IDE can confirm, Web cannot" → Check Host Strategy . Evidence: did headless Host implement hook degradation? Don't: force Modal in IM.

"OpenAPI and Web results differ" → Check Entry Convergence . Evidence: do both go through Agent front gateway into same Runtime? Don't: write second core for API.

Execution Authorization vs Host Authorization

"Who permits what" appears twice in the chain; must separate:

Execution Authorization : Can tool actually modify environment? Which paths readable? Which MCPs callable? Lands in Runtime / Sandbox / ACL .

Host Authorization : Can this entry's user start chat? View history? Who clicks confirmation? Lands in Portal Session / IDE Session / IM Identity .

Merging into one "permission" switch guarantees one of two outcomes: entry permits but execution layer has no walls; or execution layer has walls but entry pretends all smooth.

06 | Two Scenarios, Walking Through Decision Points

Scenario 1: Web Chat "Slow and Feels Like No Tools"

End-to-end: Ingress → Main Platform → Agent Front Gateway → Container Runtime → Model.

Eliminate step by step in three-layer order, don't skip steps :

First confirm if Model's fault : Run same prompt on isolated eval set. Output quality normal → Model not degraded, don't swap Model yet .

Then check Runtime three lights (Process up / Connection warm / Previous turn idle). Any red light → fix Runtime.

All three green but output still hallucinates → Only then touch Model.

Compare two approaches:

Approach A : Swap Model first. If eval set quality normal but first token still slow → Reject A , go check if connection cold or container pool empty.

Approach B : Check Runtime three lights first. Red light → fix Runtime; all green but still hallucinating → then adjust Model.

Choose B, because order cannot reverse: When Designer didn't draw wrong blueprint, swapping Designer doesn't solve "Construction Crew hasn't arrived."

Scenario 2: Same Agent, IDE Asks Human, IM Deadlocks

End-to-end: Same Runtime core, two Hosts.

Option A : Deploy separate "non-asking Agent" for IM. Short-term delivery, strategy inevitably diverges.

Option B : Keep core, add permission-event degradation for headless Hosts — turn question into text options or go automatic.

Choose B. Back to renovation analogy to close: Customer can't sign on WeChat because this storefront lacks a signature pad, not because Designer can't draw blueprints. "Whether it asks human" isn't Model's persona; it's whether Host has callback channel — degrade question to text options or auto policy, problem solved at far lower cost than maintaining second core.

07 | Selection: How Far to Self-Develop

Three-Layer Value

⚡ Quantified Efficiency : Label layers before troubleshooting; cross-team alignment shifts from "meeting shouting" to "checking table"; typical post-mortem converges from "blame Model first" to "check Contract first."

📥 Capability Sink : Product and Ops can also question with "Model / Runtime / Host" without learning internal service names.

📚 Pattern Upgrade : From "integrate one big Model" to "sign three Contracts" — upcoming topics (life of a conversation, Tool & MCP, container Runtime) all hang on these three Contracts.

Five-Dimension Selection Matrix: How Far Should You Self-Develop?

Only Model API : Chat Speed = High; Can Do Work = Low; Multi-Entry Consistency = Low; Maintenance Cost = Low; Recommended for = Demo.

Self-Developed Runtime + Official Model : Chat Speed = Medium; Can Do Work = High; Multi-Entry Consistency = Medium; Maintenance Cost = Medium; Recommended for = Strong Custom Execution Layer.

Official SDK Runtime + Self-Developed Host : Chat Speed = Medium-High; Can Do Work = High; Multi-Entry Consistency = High; Maintenance Cost = Medium; Recommended for = Default Option .

Full Self-Developed Three Layers : Chat Speed = Low; Can Do Work = High; Multi-Entry Consistency = Depends on Governance; Maintenance Cost = Extremely High; Recommended for = Very Few.

Default option is the third: core follows mature SDK, Host self-adapted. Jumping straight to full self-development of three layers is usually not a technical judgment — it's forgetting to do the math.

08 | Where It Doesn't Apply, Four Pitfalls, How to Grade Changes

Applicable / Not Applicable

Applicable : Multi-entry Agent platforms, need to align troubleshooting vocabulary; Need to split "can chat / can do / can see" for acceptance.

Not Applicable : Single-page Chat demo, no tools no workspace; Only doing Model eval leaderboard, not touching engineering chain.

Pitfall 1: Every Slowdown → Swap Model

Temptation : Model most visible, easiest to procure. Wrong Antidote : Parameter count keeps rising, bill up, first character still slow. Right Antidote : Break down first-response budget — container pool share, connection share, inference share. Model only owns its slice.

Pitfall 2: Writing Second Agent Inside Host

IDE extension ships fast, so someone embeds a second tool loop inside extension. First two months peaceful; month three Runtime changes cancellation semantics, extension's loop doesn't follow — same Agent stops clean in IDE but not on Web. Core must be one; Host only handles display, confirmation, entry identity.

Pitfall 3: Using "Frontend Permission" to Replace Execution Authorization

Temptation : Graying out button is easiest. Wrong Antidote : UI disabled, process still writes disk, still calls tools. Right Antidote : Execution authorization lands in Runtime and Sandbox; Host authorization only governs "who can start chat, who can click confirm."

Pitfall 4: Concept Learned But Not Written Into New-Hire Docs

Veterans think "isn't this common sense?" so three-term lookup table never enters team handbook. Result: new hires keep mixing terms, group argues "Model bad or Runtime bad" into perpetual motion. Enforcing "label layer first" at review time beats three post-hoc meetings.

L1 / L2 / L3 (Action Grading When Changing Concepts)

L1 : Docs / Troubleshooting vocabulary unification → Team self-decides.

L2 : Host Strategy Alignment (MCP list, degradation policies) → Cross-Entry Review.

L3 : Runtime Core or Default Model Routing Change → Platform Change Gate.

Summary

Model owns "Think", Runtime owns "Do and Stop", Host owns "See and Confirm" — three terms forbidden from impersonating each other.

Troubleshooting: Label Layer First — Spoken Phrases Must Translate to Contracts Before Action.

Execution Authorization ≠ Host Authorization — Entry Permit ≠ Process Has Walls.

Core Should Unify, Host Strategies May Differ — Headless Hosts Cannot Pretend Modals Exist.

First-Token Slow and Tool Failures → Check Runtime First, Not Procure Larger Model.

Self-Develop Default: Mature SDK for Runtime Core + Self-Developed Host Adapters.

One Brand Name Called Agent, Engineering Must Split Into At Least Three Contracts — Mixing Them Costs More Than A Slightly Weaker Model.

Follow me, James's Growth Diary, continuous sharing of practical insights to help you take fewer detours in the AI era.