K8s, Kafka, Nacos Agent Platform to Prevent Token Bankruptcy and Skill Avalanches

Problem Overview

When an LLM‑based agent is used in production, three failure modes appear: uncontrolled token consumption, unsafe tool execution, and stale or hallucinated retrieval‑augmented generation (RAG) results. These are not caused by model quality but by treating the whole agent chain as a black box without production‑grade controls.

Four‑Layer Runtime Architecture

The platform is split into four orthogonal layers, each implemented as an independent service communicating via Kafka.

Control Plane : defines which tenants can create agents, which models, skills, and knowledge domains are allowed.

Execution Plane : orchestrates the end‑to‑end task flow from request intake, retrieval, reasoning, to final response.

State Plane : persists task status, step status, idempotency keys, compensation records, and audit evidence.

Governance Plane : enforces cost limits, gray‑release, audit trails, risk interception, rollback, and stop‑loss.

Entry Layer (Agent Gateway & Task API)

The gateway performs authentication, request standardisation, generates a unique task ID and idempotency key, and pushes the task to the Kafka topic agent.tasks. It never waits synchronously for LLM, RAG, or skill execution, preventing thread‑pool exhaustion when downstream services throttle or timeout.

Orchestration Layer (Agent Orchestrator)

The orchestrator consumes tasks, builds an execution plan from the AgentSpec, advances a persistent state machine, decides whether to invoke retrieval, model inference, or a skill, records each step result and its evidence, and writes outcomes back to the state store.

Governance Services

Budget Service : implements a four‑step pre‑allocation workflow – estimate → reserve → actual → reconcile – to guarantee that token consumption never exceeds a tenant’s daily quota. The Go implementation uses a Redis Lua script to make the reservation atomic.

package budget

import (
    "context"
    "fmt"
    "time"
    "github.com/redis/go-redis/v9"
)

type Service struct { rdb *redis.Client }

type ReserveResult struct { ReservationID string; Remaining int64 }

func (s *Service) Reserve(ctx context.Context, tenantID string, estimated, hardLimit int64) (*ReserveResult, error) {
    dayKey := fmt.Sprintf("budget:%s:%s", tenantID, time.Now().Format("2006-01-02"))
    reservationID := fmt.Sprintf("%s-%d", tenantID, time.Now().UnixNano())
    script := redis.NewScript(`
local dayKey = KEYS[1]
local holdKey = KEYS[2]
local estimated = tonumber(ARGV[1])
local hardLimit = tonumber(ARGV[2])
local current = redis.call("GET", dayKey)
if not current then current = 0 else current = tonumber(current) end
if current + estimated > hardLimit then return {-1, current} end
redis.call("INCRBY", dayKey, estimated)
redis.call("HSET", holdKey, "estimated", estimated, "status", "reserved")
redis.call("EXPIRE", holdKey, 3600)
redis.call("EXPIRE", dayKey, 172800)
return {1, hardLimit - current - estimated}
`)
    holdKey := "budget_hold:" + reservationID
    values, err := script.Run(ctx, s.rdb, []string{dayKey, holdKey}, estimated, hardLimit).Int64Slice()
    if err != nil { return nil, err }
    if len(values) != 2 || values[0] < 0 { return nil, fmt.Errorf("budget exceeded") }
    return &ReserveResult{ReservationID: reservationID, Remaining: values[1]}, nil
}

func (s *Service) Finalize(ctx context.Context, reservationID string, actual int64) error {
    holdKey := "budget_hold:" + reservationID
    payload, err := s.rdb.HGetAll(ctx, holdKey).Result()
    if err != nil { return err }
    if payload["status"] != "reserved" { return nil }
    estimated := parseInt64(payload["estimated"])
    delta := estimated - actual
    // handle delta >0 (refund) or delta <0 (additional charge)
    return s.rdb.HSet(ctx, holdKey, "status", "finalized", "actual", actual).Err()
}

RAG Service : performs a two‑stage retrieval – BM25 followed by dense vector search – merges, deduplicates, applies permission and version filters, and finally reranks. Only documents with is_latest=true and a valid effective_from / effective_to window are kept.

def retrieve_evidence(query: str, tenant_id: str, now_ts: str) -> List[dict]:
    dense_hits = search_dense(query, top_k=20)
    sparse_hits = search_bm25(query, top_k=20)
    merged = merge_and_dedup(dense_hits, sparse_hits)
    visible = [hit for hit in merged
               if tenant_visible(hit["tenant_id"], tenant_id)
               and time_effective(hit["effective_from"], hit["effective_to"], now_ts)
               and hit["is_latest"]]
    reranked = rerank(query, visible)
    evidence = reranked[:5]
    if not evidence:
        raise NoEvidenceFound("no fresh evidence available")
    return evidence

Skill Gateway + Runtime Pool : each skill is described by a strict schema (name, version, risk level, timeout, concurrency limit, idempotency strategy, etc.). High‑risk skills require manual approval, have concurrency limits, and run in isolated containers or jobs.

apiVersion: agent.platform/v1
kind: Skill
metadata:
  name: restart-workload
  version: 1.4.0
spec:
  riskLevel: high
  timeoutSeconds: 20
  concurrencyLimit: 10
  idempotencyStrategy: task_id_plus_resource
  approval:
    required: true
  channels: ["ops-oncall", "change-manager"]
  inputSchema:
    type: object
    required: ["cluster", "namespace", "workload", "reason"]
  runtime:
    mode: job
    image: registry.example.com/skills/restart-workload:1.4.0

Skill Result Model

Skill implementations must return a structured result that distinguishes success, retryable failure, final failure, partial success, and waiting‑for‑approval. This enables the orchestrator to decide whether to retry, compensate, or hand off to a human.

from enum import Enum
from pydantic import BaseModel

class SkillStatus(str, Enum):
    SUCCEEDED = "SUCCEEDED"
    FAILED_RETRYABLE = "FAILED_RETRYABLE"
    FAILED_FINAL = "FAILED_FINAL"
    PARTIAL_SUCCEEDED = "PARTIAL_SUCCEEDED"
    WAITING_APPROVAL = "WAITING_APPROVAL"

class SkillResult(BaseModel):
    status: SkillStatus
    idempotency_key: str
    output: dict | None = None
    error_code: str | None = None
    error_message: str | None = None
    audit_refs: list[str] = []

State Machine and Persistence

The platform defines a persistent state machine with the following linear states, ensuring tasks can be resumed after crashes and duplicate messages are harmless.

PENDING → BUDGET_RESERVED → EVIDENCE_RETRIEVED → PLAN_GENERATED → WAITING_APPROVAL → SKILL_EXECUTING → COMPLETED → FAILED → COMPENSATING

Two relational tables store the task and step information. Unique constraints on idempotency_key (task level) and step_seq (step level) guarantee idempotent processing.

CREATE TABLE agent_task (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    task_id VARCHAR(64) NOT NULL UNIQUE,
    tenant_id VARCHAR(64) NOT NULL,
    agent_name VARCHAR(128) NOT NULL,
    spec_version VARCHAR(32) NOT NULL,
    idempotency_key VARCHAR(128) NOT NULL,
    status VARCHAR(32) NOT NULL,
    budget_reservation_id VARCHAR(64),
    risk_level VARCHAR(16) NOT NULL,
    created_at DATETIME NOT NULL,
    updated_at DATETIME NOT NULL,
    UNIQUE KEY uk_tenant_idempotency (tenant_id, idempotency_key),
    KEY idx_tenant_status (tenant_id, status)
);

CREATE TABLE agent_task_step (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    task_id VARCHAR(64) NOT NULL,
    step_seq INT NOT NULL,
    step_type VARCHAR(32) NOT NULL,
    step_status VARCHAR(32) NOT NULL,
    request_payload JSON NOT NULL,
    response_payload JSON,
    error_code VARCHAR(64),
    started_at DATETIME NOT NULL,
    finished_at DATETIME,
    UNIQUE KEY uk_task_step (task_id, step_seq),
    KEY idx_task_type (task_id, step_type)
);

Kafka Topics

agent.tasks

: main task flow agent.task-retry: retryable tasks agent.task-dlq: dead‑letter tasks agent.approvals: human‑approval events agent.audit: audit and governance events

Kafka decouples the synchronous request path, absorbs traffic spikes, and provides reliable replay for audit and compensation.

Governance Rules (Spec)

Agent behaviour is versioned in an AgentSpec YAML stored in Git. The spec contains model routing, budget policies, RAG policies, skill allow‑list, human‑approval requirements, and risk patterns. CI validates the spec, runs regression tests, and performs shadow execution on real traffic without side effects.

apiVersion: agent.platform/v1
kind: AgentSpec
metadata:
  name: sre-incident-responder
  version: 2.3.1
spec:
  modelPolicy:
    primary: gpt-4.1
    fallback: [claude-sonnet-4, qwen2.5-72b-instruct]
  maxContextTokens: 32000
  budgetPolicy:
    dailyTokenLimit: 1000000
    softLimitRatio: 0.85
    overdraftRatio: 0.05
  ragPolicy:
    knowledgeBase: sre-runbooks
    topK: 5
    requireCitation: true
    freshnessWindowMinutes: 1440
  skillPolicy:
    allow: [query-metrics, restart-workload, create-ticket]
  humanApprovalRequired: [restart-workload]
  riskPolicy:
    forbiddenPatterns: ["DROP DATABASE", "DELETE FROM .*", "rm -rf /"]

The spec drives gray‑release, budget enforcement, citation requirements, and risk‑based routing.

Skill Definition Details

A minimal production‑grade skill must declare at least five fields: name: identifier used by agents version: supports gray‑release and rollback risk_level: low / medium / high, determines if manual approval is required timeout_seconds: prevents indefinite hangs idempotency_strategy: defines how duplicate requests are deduplicated (e.g., task‑id‑plus‑resource)

Additional optional fields include concurrencyLimit, approval, inputSchema, and runtime (container image, job mode).

RAG Pipeline with Freshness and Permission Filters

The retrieval chain consists of query rewrite, BM25 recall, dense recall, permission filtering, version/valid‑time filtering, reranking, and finally evidence verification. Only chunks that satisfy all three conditions are considered:

is_latest=true

effective_from ≤ now < effective_to

Tenant has the required acl_tags If no trustworthy evidence is found, the task is rejected before any skill execution.

Document Schema for RAG

{
  "doc_id": "runbook-payment-rollback",
  "chunk_id": "runbook-payment-rollback#section-3",
  "tenant_id": "infra",
  "source_type": "runbook",
  "version": "2026-07-10.2",
  "effective_from": "2026-07-10T21:00:00Z",
  "effective_to": null,
  "is_latest": true,
  "acl_tags": ["sre", "payment"],
  "content": "当支付服务出现连接池耗尽时，先查看...",
  "embedding": []
}

These fields enable the RAG service to filter out stale or unauthorized knowledge at query time.

Answer Verification

After the LLM generates an answer, the platform checks that the answer includes cited chunk IDs and that those chunks actually support the asserted conclusion. If citations are missing or insufficient, automatic execution of high‑risk skills is blocked.

Execution Isolation with Kubernetes

Kubernetes provides isolated execution environments for the orchestrator, retrieval service, and each high‑risk skill (as Deployments or Jobs). This allows independent scaling, resource limits, and graceful shutdown without affecting other components.

Dynamic Governance with Nacos

Nacos stores dynamic configuration and service discovery:

Service discovery for model gateways and skill instances

Runtime configuration such as token limits, skill concurrency caps, and feature flags

Gray‑release routing rules (e.g., enable a new spec version only for a subset of tenants)

Changes in Nacos take effect within seconds, allowing emergency shutdown of a risky skill without redeploying code.

Release Process and Rollback

Agent behaviour is versioned in Git. CI runs static validation, regression tests, and shadow execution on live traffic. Gray‑release is driven by risk level: P0 (critical): always allowed P1 (operational): limited to read‑only actions P2 (reporting): delayed or queued when budget is tight P3 (batch analysis): rejected if budget exhausted

If a new version causes failures, the platform can instantly roll back by switching the spec version in Git and updating Nacos routing.

Applicability

The architecture shines in multi‑tenant, high‑throughput environments where agents invoke real‑world tools, knowledge bases evolve rapidly, and strict audit, rollback, and gray‑release capabilities are required. It is overkill for single‑tenant, low‑frequency, read‑only assistants.

Pre‑Launch Checklist (12 Items)

Token pre‑allocation per tenant is in place.

Separate budget strategies for high‑priority (P0) and low‑priority (P2‑P3) tasks.

Each skill defines timeout, concurrency limit, risk level, and idempotency strategy.

High‑risk skills can be disabled instantly via Nacos.

RAG retrieval includes permission, version, and time‑window filters.

Critical conclusions must carry citation evidence; lack of evidence blocks auto‑execution.

Task and step states are persisted for crash recovery.

Kafka topics are split for main flow, retries, dead‑letter, and approvals.

AgentSpec is version‑controlled in Git with rollback capability.

New versions undergo shadow execution before production writes.

Audit logs can reconstruct "who invoked which skill with which spec version".

Clear degradation paths exist for budget exhaustion, model 429, missing knowledge, or partial skill success.