From Demo to Production: Key Practices for Engineering LLM Agents

One‑Sentence Conclusion

Agent engineering is not just packaging a script into Docker — it is about turning modules 1‑11 into a maintainable service contract that defines interfaces, externalized configuration, session persistence, rate limiting, and deployment so that the demo runs on a developer laptop while the production service runs on a rollback‑able, observable infrastructure.

Problem Scenario

When modules 1‑11 are assembled the agent can converse, call tools, pass security checks, emit traces, and run golden regressions. Running python main.py works locally, but switching to an HTTP service with multiple users 24/7 exposes engineering‑layer issues:

Wrong interface shape : Long ReAct tasks over synchronous HTTP hit 60 s timeouts, return 504, and cause duplicate writes (module 5).

Hard‑coded keys and config : OPENAI_API_KEY baked into the image, staging accidentally points to production vector store, and changing max_steps requires a code release (module 10).

Session state stored in process : After a HITL pause the user refreshes, the checkpoint is missing on another pod, and multi‑turn dialogs lose context (module 6).

Cost and concurrency out of control : Promotional spikes generate 100 concurrent runs, exhausting LLM quota; a tenant runs unlimited ReAct loops, causing a night‑time bill surge (modules 2, 11).

Trace and evaluation not in CI : JSONL traces pile up in /tmp, pods are recreated and evidence is lost; after a prompt change no regression is run (module 11).

Deployment like a monolithic script : No health check, graceful shutdown, or version tag, leading to mismatched old/new agent_version behavior during rolling updates.

Core Concept 1: Interface Shape – Sync, Stream, Async, Queue

Agent runs can last seconds to minutes (multi‑step ReAct + RAG + HITL). The interface must match the expected latency and side‑effects; a single POST JSON endpoint is insufficient.

Common Interface Comparison

CLI : Ideal for dev debugging, batch jobs, internal ops. Zero gateway, easy scripting. Not suitable for multi‑tenant or HITL UI.

Synchronous HTTP : Fits single‑turn FAQ, ≤1 tool, short context. Simple implementation but long tasks timeout and block connections.

SSE / Streaming HTTP : Streams token‑by‑token and tool progress. Improves UX but requires a structured event schema ( text, tool_call_started, tool_call_ended, waiting_hitl, completed).

WebSocket : Enables bidirectional mid‑run changes, cancellation, extra parameters. Natural for HITL but adds connection management, sticky‑session, and scaling complexity.

Queue Worker : Decouples long‑running reports, batch evaluation, offline tasks. Results must be fetched via polling or webhook.

Design Principles

Expose a first‑class run_id (or trace_id) for status lookup, cancellation, and trace correlation.

Runs longer than ~30 s should return 202 Accepted with run_id and be followed via SSE/WebSocket/poll; never block a sync connection.

Streaming events must be structured: emit tool_call_started, observation, waiting_hitl, completed so front‑end and module 11 share the same schema.

Clients retry with an Idempotency-Key (usually session_id + client_message_id) to avoid duplicate tool executions.

Trade‑off : WebSocket gives the best UX but is the most expensive to operate; most B‑side internal agents are fine with HTTP + SSE. Queues suit tasks that do not need immediate replies and should be split from the conversational agent to avoid worker backlog.

Core Concept 2: Configuration & Environment – Keys, Routing, Switches

In the demo MODEL = "gpt-4o" and MAX_STEPS = 10 are hard‑coded. In production the same image runs dev / staging / prod, and all differences must be externalized.

Configuration Layers

Keys (API keys, DB connection strings) – low change frequency – store in Secret Manager or K8s Secret.

Environment Constants (region, log level, trace sampling) – low frequency – store as environment variables.

Agent Behavior ( max_steps, model id, temperature, tool whitelist) – medium frequency – store in config.yaml with hot‑load or release.

Security Policy (T2 tool list, HITL rules, PII rules) – medium frequency – store in policy.yaml with a policy_version field.

Feature Flags (enable RAG, reflection, new planner) – high frequency – use a feature‑flag service or config centre.

Model routing is externalized in config/models.yaml. Example:

# config/models.yaml (excerpt)
routes:
  default: {provider: openai, model: gpt-4o-mini, temperature: 0.2}
  planning: {provider: openai, model: o1-mini, temperature: null}
  fallback: {provider: azure, model: gpt-4o-mini}

rules:
- when: {step_type: plan, token_estimate_gt: 8000}
  use: planning
- when: {error_count_gte: 2}
  use: default
- when: {provider_health: degraded}
  use: fallback

The agent_version (module 10, 11) should be bound to each run as git_sha + config_hash + policy_version so that debugging can reproduce the exact configuration.

Environment Isolation Checklist

Staging must never connect to production vector or memory stores.

Secrets never go into the image or git; .env is for local development only.

Configuration changes are auditable; production max_steps changes go through PR + regression (module 11), not direct SSH edits.

Trade‑off : A config centre adds dependency and latency; small teams may use versioned YAML + env vars but must keep secrets separate from behavior config.

Core Concept 3: Session & State – Stateless Instances + External Persistence

Module 5’s HITL requires checkpoint persistence; module 2’s messages cannot live only in process memory. The default engineering posture is to externalize both.

What to Store

session:{id}

– message summary or full history – TTL 7‑90 days per product. checkpoint:{run_id} – orchestration state (step, plan, scratchpad, pending tool calls) – TTL = HITL wait period + buffer. approval:{token} – tool + params digest – TTL ≈ 15 minutes.

Session Stickiness

Ideal: any instance reads the same session store → no gateway stickiness needed.

If a local cache is used, enforce sticky sessions or a distributed lock; otherwise pod switches lose state.

During scaling, new instances only read storage; they do not assume prior runs.

Trade‑off : Storing all messages in Redis is expensive; a hot‑data Redis + cold object‑storage pattern works with module 2’s compact strategy. Checkpoints should be small; large tool results are stored as blobs outside (module 11).

Core Concept 4: Cost, Rate Limiting & Degradation – Agent‑Specific Traffic Governance

Beyond ordinary API QPS limits, agents need token, step, tool‑side‑effect, and tenant‑budget limits.

Multi‑Level Quotas

Gateway : IP / API‑Key QPS (e.g., 100 req/min).

Tenant : Concurrent runs and daily token budget (e.g., 10 concurrent, 500 k tokens/day).

Single Run : max_steps, wall‑clock, per‑run token limits – configurable termination table (module 5).

Tool : T2 call count, HTTP tool egress – tier limits defined in module 10.

Degradation Strategies

If tenant token budget is exhausted → reject new runs or allow read‑only tools only.

If LLM returns 429 or times out → route to a fallback model (configurable).

If step count approaches limit → orchestrator forces finalize and explains incompleteness (module 6).

If RAG latency is too high → skip rerank, reduce top_k (module 8).

Caching

Embedding / retrieval: short‑TTL cache keyed by query + index version, tenant‑isolated.

LLM completions: cache only stable system prompts and document chunk summaries; user‑changed inputs miss the cache.

Tools: read‑only, idempotent tools (weather, exchange rate) can be cached; write tools must never be cached.

Trade‑off : Aggressive degradation saves cost but hurts completion rate; module 11 monitors both cost and completion to adjust thresholds.

Core Concept 5: Deployment, Trace Storage & Release – Hooking Observability into Ops

Deployment Baseline

Process model : A single run is CPU‑intensive + I/O‑wait; set Uvicorn/Gunicorn worker count based on LLM concurrency, not CPU cores.

Health checks : /health for liveness; /ready must verify key read, session store, and vector store connectivity.

Graceful shutdown : On SIGTERM stop accepting new runs; in‑flight runs must finish or checkpoint.

Image : Multi‑stage build; exclude secrets and evaluation datasets.

Horizontal scaling : Stateless instances; external session/checkpoint storage.

Trace & Evaluation Engineering (module 11)

Two pipelines are defined:

Runtime trace collection : Each run tags agent_version, stores the trace in a backend, and enables version‑to‑trace correlation.

Pre‑release golden regression : CI runs a P0 golden subset; nightly runs the full suite with cost stats. Failures block the release and produce a report with completion‑rate delta and degraded trace_id (via compare_versions).

Sampling strategy: 100 % for error runs, T2 tools, and new models; low sampling for simple FAQs. Sampling rates are configurable without code changes.

Release & Rollback

Each release tags agent_version and uses K8s RollingUpdate while retaining the previous version for quick rollback.

Configuration and code are decoupled; rolling back code does not automatically roll back policy – a matrix test is required.

Canary: route traffic by tenant_id or percentage to the new version and compare completion, latency, and cost (module 11 dashboard).

Trade‑off : OpenTelemetry provides a standard trace schema but is costly to integrate; early stages may use JSONL → object storage with a simple query schema, but the field schema should be fixed once to avoid later migrations.

Design Checklist for Agent Production Readiness

Interface shape (sync / SSE / WebSocket / queue) defined and long tasks never block sync connections.

Expose stable run_id / trace_id supporting cancel, resume, and trace lookup.

Clients retry with Idempotency-Key; T2 tools are idempotent.

Secrets separated from behavior config and never baked into images or git. agent_version includes code, config, and policy hashes and is recorded per run.

Model routing, max_steps, and sampling rates are externalized and isolated between staging and prod.

Session and checkpoint stored externally; instances remain stateless; HITL can resume on any instance.

Tenant‑level concurrency, token, and daily budget limits are observable with rejection reasons.

LLM timeouts trigger fallback routing; step limits trigger forced finalize. /health and /ready cover dependency connectivity; SIGTERM leads to graceful shutdown.

Trace uses tiered sampling and hot/cold storage; pod recreation does not lose critical trace data.

Release pipeline integrates P0 golden gate and compare_versions reporting.

Rollback by agent_version and canary deployment capabilities are present.

Pseudo‑code / Minimal Service Skeleton (framework‑agnostic)

# --- Configuration loading (once at startup, or watch for hot‑reload) ---

from dataclasses import dataclass

@dataclass
class AppConfig:
    agent_version: str          # git_sha + config_hash
    models: ModelRouterConfig   # from config/models.yaml
    orchestrator: OrchestratorConfig  # max_steps, timeouts
    policy: PolicyConfig        # from policy.yaml
    trace: TraceExportConfig    # sampling rate, backend URL

def load_config(env: str) -> AppConfig:
    # Secrets injected from environment / Secret Manager, never in plain YAML
    ...

# --- HTTP entry point (sync short tasks + async long tasks) ---

class AgentService:
    def __init__(self, cfg, session_store, trace_exporter, orchestrator):
        self.cfg = cfg
        self.sessions = session_store
        self.trace = trace_exporter
        self.orchestrator = orchestrator

    async def create_run(self, tenant_id, session_id, message, idempotency_key=None):
        # Quota check (tenant concurrency, daily budget)
        await self.quota.check(tenant_id)
        if idempotency_key:
            existing = await self.sessions.get_idempotent(idempotency_key)
            if existing:
                return existing  # avoid duplicate side‑effects
        ctx = await self.sessions.load_or_create(session_id, tenant_id)
        ctx.agent_version = self.cfg.agent_version
        if self._estimate_duration(ctx, message) > SHORT_RUN_THRESHOLD:
            run_id = await self._enqueue_background(ctx, message)
            return {"run_id": run_id, "status": "accepted"}
        return await self._run_sync(ctx, message)

    async def _run_sync(self, ctx, message):
        run_id = new_run_id()
        with self.trace.begin_run(run_id, ctx):
            result = await self.orchestrator.run(ctx, message)
        if result.status == "waiting_hitl":
            await self.sessions.save_checkpoint(run_id, ctx.snapshot())
            await self.sessions.save_messages(ctx)
        return {"run_id": run_id, "status": result.status, "output": result.text}

    async def resume_run(self, run_id, approval_token, user_decision):
        ctx = await self.sessions.load_checkpoint(run_id)
        # Verify approval_token (module 10) and continue execution
        ...

# --- Model router (module 1 + engineering config) ---

class ModelRouter:
    def resolve(self, ctx, step_type: str) -> ModelEndpoint:
        if ctx.error_count >= 2:
            return self.config.routes["fallback"]
        if step_type == "plan":
            return self.config.routes["planning"]
        return self.config.routes["default"]

# --- Brain uses the router (module 1) ---

class Brain:
    def decide(self, ctx, messages, tools):
        endpoint = self.router.resolve(ctx, step_type=ctx.current_step_type)
        return self.providers[endpoint.provider].complete(
            model=endpoint.model,
            messages=messages,
            tools=tools,
            temperature=endpoint.temperature,
        )

Mapping to a Real‑World Project (Practical Series 8)

CLI → HTTP wrapper (FastAPI /runs endpoint).

External config via config.yaml and environment variables.

ModelRouter connects to the Brain component.

Session/checkpoint storage using Redis or SQLite for a start.

SSE streaming for tool progress and HITL states (optional).

Trace export and sampling integrated with Practical 7 JSONL or OpenTelemetry.

CI golden gate using GitHub Actions.

Dockerfile + compose with health checks.

Tenant‑level rate‑limit skeleton ready for extension.

Summary

Agent engineering centers on a service contract: match interface shape to model latency, expose run_id for state and trace.

Externalize keys, model routing, policies, and feature flags; bind them with agent_version for reproducible regression and rollback.

Stateless instances with external session and checkpoint storage enable multi‑instance HITL and approval flows.

Rate limiting and degradation must cover tenant budgets, step limits, and fallback models, all driven by a single configuration source.

Trace storage and CI gate are the production form of module 11: tiered sampling, hot/cold layers, and blocking bad releases.

The watershed from demo to service is not Docker but a configurable, extensible, rollback‑able, testable deployment layer that can handle real traffic.