Vertical Agent Deployment: 4-Layer Architecture, 7-Step Checklist & Local 70B Cost Analysis

Vertical Agent ≠ Chatbot Wrapper

Three non-negotiable boundaries separate a vertical agent from a generic chatbot:

Bounded domain : The agent handles only one vertical (e.g., labor-contract review) and rejects out-of-scope requests.

Verifiable acceptance : Output must be judgeable by rules or human spot-checks (risk checklist, pass/fail), not subjective "pleasant conversation".

Tool use : The agent calls APIs, reads files, queries databases, and writes results — it is not pure dialogue.

While chatbots sell "conversation time", vertical agents sell "task completion rate" — a fundamental valuation difference that enables outcome-based pricing.

Four-Layer Architecture

Access layer : Normalizes user/file/API inputs (PDF, Word, web → plain text).

Orchestration layer : The control-loop code that decides how many model calls and in what order. As argued in the Memdoor article, orchestration > scale — a 27B model with good orchestration can approach frontier single-call performance.

Model layer : Runs local 27B/70B via MLX or Ollama, with cloud API fallback for long-tail cases.

Tool layer : Extraction, retrieval, verification, write-back — domain knowledge hardened into callable functions; this is the moat.

Memory layer : Vector store + session state so the agent remembers per-client special rules across requests.

Tech Selection: Local vs. Cloud

Core judgment: once a vertical agent is proven and volume grows, local deployment delivers deterministic cost reduction + privacy simultaneously . Model weights are commoditizing (GPT-6.1 Terra at $1/$5 per 1M tokens, Qwen4 27B expected single-GPU), so leverage stays forever in "outside the model" — your domain data and closed-loop acceptance.

Runnable Example: Local Contract-Review Agent

Scenario : Law firm / in-house counsel; contracts contain trade secrets, cannot leave premises. M5 Max 128GB runs Qwen3.8-27B at 18–32 tok/s (64GB MacBook Pro also works, slower).

Decomposition : Access (PDF/Word → txt) → Orchestration (extract clauses first, then review clause-by-clause) → Model (local 27B, temperature=0) → Tools (rule-library verification) → Memory (client historical risk library).

Two critical insights:

Long context is a silent killer : Don't stuff 128K tokens upfront. Extract first, then review, keeping context ≤32K — latency and cost drop an order of magnitude.

Closed-loop acceptance : Force the model to output only JSON (risks list + level + suggestion), then validate JSON schema with rules — this is "verifiable acceptance" and the prerequisite for measuring failure rates later.

Zero-to-One Seven-Step Checklist

Step 6 is the most important: sample 50 documents, record "missed" and "false positive" rates — don't just watch it run . The Memdoor article showed that 9.8 vs 8.0 scores are mostly noise; your agent's completion rate must be trusted via sampled, acceptance-gated evaluation.

Cost Accounting: Local vs. Cloud

Local : M5 Max 128GB ≈ ¥47,000 one-time + negligible electricity; marginal cost per call ≈ 0; data never leaves.

Cloud GPT-6.1 Terra : 10K input + 2K output ≈ $0.02 ≈ ¥0.14 per call; 1,000 calls/month ≈ ¥140, 10,000 calls/month ≈ ¥1,400.

Break-even : At ¥0.14/call and hardware monthly depreciation ≈ ¥1,500, local becomes cheaper at > ~11,000 calls/month — and compliance cost of data egress is extra.

Conclusion is blunt: for high-volume vertical agents, local deployment isn't sentiment — it's dictated by the ledger .

Three Fatal Pitfalls

Vague acceptance criteria : "Reviewed well" cannot be optimized; must be codified as machine-judgable standards (JSON schema / pass rate).

Only tuning the model, not the flow : A small orchestration tweak lets 27B approach frontier single-call — fix the flow first, swap the model later.

Over-dependence on cloud APIs : Betting all logic on someone else's API makes cost explode linearly with usage and forces data egress — the vertical agent's moat is precisely "local + domain data".