Building a Scalable Smart Tag System for 100k QPS with AI‑Generated Code

1. The problem is not "can we tag" but "can tags enter the main pipeline"

Many teams discuss "AI tag systems" by emphasizing model performance, yet the first issues in production are not label accuracy but write spikes, amplified timeouts, duplicate tagging, cost loss of control, and inconsistent states.

This article does not cover how to call a large model in a demo; instead it examines a realistic flow where a content platform continuously writes products, posts, comments and activity assets, and recommendation, search, risk control and operations all depend on tag results. The tag system must produce stable outputs under high concurrency without propagating the uncertainty of large models to the main business chain.

The 100 k QPS mentioned is an architectural design goal, not a measured result. Wherever real load‑test data is missing, the article explicitly marks the statements as engineering goals, capacity estimates, or suggested testing methods.

2. AI is better suited as a co‑pilot than as the architect

The system can let AI generate a lot of code, but only after clear boundaries are defined. Work is split into two layers:

AI‑generated parts: interface skeletons, DTOs, Proto files, deployment YAML, test scripts, monitoring panel templates, basic exception branches, repetitive CRUD.

Human‑decided parts: tag state model, sync/async switch points, idempotency semantics, degradation strategies, cache consistency, cost caps, replay and audit rules.

If boundaries are not clear, AI may produce a runnable service that lacks essential production constraints, such as how to compensate after a timeout or which lock‑failure response to return.

3. View the tag system as five planes: Access, Decision, Execution, State, Governance

The architecture can be broken down into five planes instead of a simple component list like "Spring Boot + Kafka + Redis + LLM Gateway".

┌──────────────────────────┐
               │      Governance Plane     │
               │ Quota/Rate‑limit/Audit/Replay/Canary │
               └────────────┬─────────────┘
                            │
   ┌────────────┐   ┌──────▼───────┐   ┌────────────────┐
   │ Access Plane│──▶│ Decision Plane│──▶│ Execution Plane│
   │ API/GRPC   │   │ Sync/Async/Model Routing│ │ LLM/Rule/Summarizer│
   └────────────┘   └──────┬───────┘   └───────┬───────┘
                            │               │
                            ▼               ▼
               ┌────────────────┐   ┌────────────────┐
               │   State Plane  │   │   Event Channel│
               │ Cache/Version/Idempotency│ │ Kafka/Retry/DLQ│
               └────────────────┘   └────────────────┘

3.1 Access Plane

Handles request reception, authentication, traffic shaping, parameter validation and tenant isolation. Its goal is to reject illegal requests early and propagate context such as tenant ID, content type and priority downstream.

3.2 Decision Plane

Determines whether a request should be processed synchronously or queued asynchronously, whether to use rule‑based output, a lightweight model, or a full‑scale model, the timeout, cost budget and acceptable degradation level. Without this plane, the system would default to calling the model for every request.

3.3 Execution Plane

Executes rule engines, summarizers, embedding services and large‑model calls. It should not make business state judgments; it only runs tasks under given strategies and returns results.

3.4 State Plane

Tracks the evolving state of each piece of content:

Current content version.

Which tag version is active.

Whether the tag was produced synchronously or as an async compensation.

Status values: PENDING, RUNNING, SUCCEEDED, FAILED, DEGRADED.

Retry count and dead‑letter status.

3.5 Governance Plane

Turns the system from "can work" to "can be operated":

Model call quotas and tenant budgets.

Prompt version canary and rollback.

Dead‑letter replay and manual repair.

Hot‑content protection.

Audit logs and traceability.

4. Synchronous return is not the default but a limited capability

Many teams default to returning tags immediately after a call, which works at low traffic but should be treated as a constrained option under high concurrency. The recommended approach is to assign service levels per request type:

Request Type   | Typical Source          | Return Requirement | Recommended Strategy
----------------|--------------------------|--------------------|----------------------
Strong real‑time| Product publish, audit   | 100 ms‑to‑seconds   | Rule first, model second; sync preferred, timeout → async
Near real‑time | Recommendation feature fill| Seconds‑to‑minutes| Write event, async generate, result back‑fill
Offline refresh| Historical content recompute| Hours              | Batch processing, allow long latency, emphasize throughput cost
Hotspot re‑check| Repeated request for same content| Immediate | Prioritize cache or latest version

This table turns a vague latency goal into concrete architectural choices.

5. Define a data model so the system has consistent state semantics

Without a stable data model, code alone cannot be governed. The core tables are:

5.1 Content Tag Snapshot Table

CREATE TABLE content_label_snapshot (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    tenant_id BIGINT NOT NULL,
    content_id VARCHAR(64) NOT NULL,
    content_version BIGINT NOT NULL,
    label_version BIGINT NOT NULL,
    label_payload JSON NOT NULL,
    source_type VARCHAR(16) NOT NULL,
    model_name VARCHAR(64),
    prompt_version VARCHAR(32),
    status VARCHAR(16) NOT NULL,
    generated_at DATETIME NOT NULL,
    created_at DATETIME NOT NULL,
    UNIQUE KEY uk_content_label_ver (tenant_id, content_id, content_version, label_version),
    KEY idx_content_latest (tenant_id, content_id, generated_at)
);

The snapshot stores the result of a specific content version under a specific label version, enabling replay and audit.

5.2 Tag Task Table

CREATE TABLE label_task (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    tenant_id BIGINT NOT NULL,
    request_id VARCHAR(64) NOT NULL,
    content_id VARCHAR(64) NOT NULL,
    content_version BIGINT NOT NULL,
    channel VARCHAR(16) NOT NULL,
    priority INT NOT NULL DEFAULT 0,
    status VARCHAR(16) NOT NULL,
    retry_count INT NOT NULL DEFAULT 0,
    next_retry_at DATETIME DEFAULT NULL,
    last_error_code VARCHAR(64) DEFAULT NULL,
    last_error_message VARCHAR(512) DEFAULT NULL,
    created_at DATETIME NOT NULL,
    updated_at DATETIME NOT NULL,
    UNIQUE KEY uk_request_id (tenant_id, request_id),
    KEY idx_pending_retry (status, next_retry_at)
);

The task table describes the processing workflow, while the snapshot table records the outcome. Separating them clarifies whether a task failed or whether a usable tag exists.

5.3 Cost & Quota Table

CREATE TABLE label_quota_usage (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    tenant_id BIGINT NOT NULL,
    biz_date DATE NOT NULL,
    model_name VARCHAR(64) NOT NULL,
    request_count BIGINT NOT NULL DEFAULT 0,
    input_tokens BIGINT NOT NULL DEFAULT 0,
    output_tokens BIGINT NOT NULL DEFAULT 0,
    degraded_count BIGINT NOT NULL DEFAULT 0,
    UNIQUE KEY uk_tenant_day_model (tenant_id, biz_date, model_name)
);

Many systems monitor cost only via logs, which makes troubleshooting hard. Treating quota and cost as first‑class data lets the system enforce limits in real time.

6. Both happy and failure paths must be fully implemented

6.1 Normal Path

Request → Parameter validation/auth → Read content version & summary → Query cache & latest snapshot → Decision sync/async → Rule/cache hit → Return
Otherwise → Call LLM Gateway → Write tag snapshot → Publish tag change event → Return result

6.2 Exception Path

Model timeout / rate‑limit / budget exhausted → Mark sync failure → Write label_task status → Send Kafka async compensation event → Return old or degraded tag → Consumer retries → If max retries → DLQ → Manual replay or auto reprocess

Both paths are required; a system that only implements the happy path will show gray states under peak load.

7. Core Java service focuses on state transition, not model calls

The following code snippet shows the critical synchronous entry point, solving three problems:

Prevent duplicate model calls for the same content.

Avoid blocking the main pipeline after a sync timeout.

Ensure async compensation is reliably dispatched even when degraded.

@Service
public class LabelCommandService {
    private final LabelSnapshotRepository snapshotRepository;
    private final LabelTaskRepository taskRepository;
    private final LabelDecisionEngine decisionEngine;
    private final LlmGatewayClient llmGatewayClient;
    private final KafkaTemplate<String, LabelAsyncEvent> kafkaTemplate;
    private final RedisTemplate<String, String> redisTemplate;

    public LabelResponse generate(LabelRequest request) {
        String idemKey = "label:req:" + request.getTenantId() + ":" + request.getRequestId();
        Boolean firstSeen = redisTemplate.opsForValue()
                .setIfAbsent(idemKey, "1", Duration.ofMinutes(10));
        if (Boolean.FALSE.equals(firstSeen)) {
            return LabelResponse.accepted("DUPLICATE_REQUEST");
        }
        LabelTask task = taskRepository.initTask(request);
        LabelDecision decision = decisionEngine.decide(request);
        if (decision.skipModel()) {
            LabelSnapshot snapshot = snapshotRepository.save(decision.toRuleSnapshot());
            return LabelResponse.success(snapshot, "RULE_DIRECT");
        }
        String contentLockKey = "label:lock:" + request.getTenantId() + ":" + request.getContentId();
        Boolean lockAcquired = redisTemplate.opsForValue()
                .setIfAbsent(contentLockKey, request.getRequestId(), Duration.ofSeconds(5));
        if (Boolean.FALSE.equals(lockAcquired)) {
            publishAsyncTask(task, "LOCK_BUSY");
            return LabelResponse.accepted("ASYNC_FALLBACK");
        }
        try {
            LlmLabelResult llmResult = llmGatewayClient.generate(
                    request,
                    decision.modelPlan(),
                    Duration.ofMillis(decision.syncTimeoutMillis()));
            LabelSnapshot snapshot = snapshotRepository.save(
                    LabelSnapshot.fromLlmResult(request, llmResult, LabelStatus.SUCCEEDED));
            taskRepository.markSucceeded(task.getId(), snapshot.getId());
            return LabelResponse.success(snapshot, "SYNC_LLM");
        } catch (LlmTimeoutException | LlmRateLimitException ex) {
            taskRepository.markDegraded(task.getId(), ex.getCode(), ex.getMessage());
            publishAsyncTask(task, ex.getCode());
            LabelSnapshot latest = snapshotRepository.findLatestAvailable(request.getTenantId(), request.getContentId());
            if (latest != null) {
                return LabelResponse.degraded(latest, "LAST_GOOD_SNAPSHOT");
            }
            return LabelResponse.accepted("ASYNC_ONLY");
        } finally {
            redisTemplate.delete(contentLockKey);
        }
    }

    private void publishAsyncTask(LabelTask task, String reasonCode) {
        kafkaTemplate.send(
                "label.async.generate",
                task.getTenantId() + ":" + task.getContentId(),
                new LabelAsyncEvent(task.getId(), reasonCode));
    }
}

Key points: requestId is the idempotency key, distinct from the content lock.

Failure semantics are recorded in task status, not just logged.

Degraded responses are explicitly categorized as LAST_GOOD_SNAPSHOT, ASYNC_ONLY or RULE_DIRECT.

8. LLM Gateway is a control layer, not just an SDK wrapper

Direct SDK integration leads to scattered model‑switching, prompt‑canary and retry logic. A dedicated gateway centralizes:

Prompt template loading and versioning.

Model routing and fallback order.

Token budget checks and truncation.

Latency, error‑code and cost reporting.

Unified response structure and error semantics.

class ModelPlan:
    model_name: str
    max_input_tokens: int
    max_output_tokens: int
    timeout_ms: int
    prompt_version: str

class LabelGatewayService:
    def __init__(self, prompt_manager, tokenizer, quota_guard, model_router, llm_client):
        self.prompt_manager = prompt_manager
        self.tokenizer = tokenizer
        self.quota_guard = quota_guard
        self.model_router = model_router
        self.llm_client = llm_client

    async def generate_labels(self, req):
        prompt = self.prompt_manager.resolve(req.biz_type, req.tenant_id)
        plan = self.model_router.route(req.biz_type, len(req.content), req.priority)
        estimated_tokens = self.tokenizer.estimate(prompt, req.content)
        if estimated_tokens > plan.max_input_tokens:
            req = req.with_compressed_content()
            estimated_tokens = self.tokenizer.estimate(prompt, req.content)
        self.quota_guard.assert_allowed(
            tenant_id=req.tenant_id,
            model_name=plan.model_name,
            estimated_tokens=estimated_tokens
        )
        return await self.llm_client.generate(
            model=plan.model_name,
            prompt_version=plan.prompt_version,
            prompt=prompt,
            content=req.content,
            timeout_ms=plan.timeout_ms,
            max_output_tokens=plan.max_output_tokens,
        )

The order matters: model routing → budget check → token estimation → possible compression → execution.

9. Asynchronous compensation must guarantee eventual consistency, not infinite retries

Many articles stop at "Kafka sends a failure message". A robust async consumer must:

Check task status using taskId to avoid duplicate consumption.

If the content version has advanced, cancel or replay the stale task.

Re‑validate quota before model call because system state may have changed.

On success, write a new snapshot and update task status.

On failure, distinguish immediate retry, delayed retry or dead‑letter based on error type.

@KafkaListener(topics = "label.async.generate", groupId = "label-consumer")
public void consume(LabelAsyncEvent event) {
    LabelTask task = taskRepository.findById(event.taskId());
    if (task == null || task.isTerminal()) return;
    if (taskRepository.isStaleVersion(task.getTenantId(), task.getContentId(), task.getContentVersion())) {
        taskRepository.markCanceled(task.getId(), "STALE_CONTENT_VERSION");
        return;
    }
    try {
        LabelSnapshot snapshot = asyncLabelExecutor.execute(task);
        taskRepository.markSucceeded(task.getId(), snapshot.getId());
    } catch (RetryableException ex) {
        taskRepository.scheduleRetry(task.getId(), ex.getCode(), nextRetryAt(task.getRetryCount()));
        throw ex;
    } catch (NonRetryableException ex) {
        taskRepository.markDeadLetter(task.getId(), ex.getCode(), ex.getMessage());
    }
}

Important notes:

"Can retry" does not mean "retry forever"; budget exhaustion or model limits should stop further attempts.

Stale content versions must be cancelled to avoid overwriting newer tags.

10. Scaling from a single node to a distributed system should be incremental, based on failure points

Typical stages:

Stage 1 – Direct model call: Low traffic, few content types, strong real‑time needs, manual oversight.

Stage 2 – Cache + async compensation: Triggered by duplicate content, model latency variance, emerging timeouts. Adds Redis cache, task table, Kafka async fallback, snapshot replay.

Stage 3 – Independent model gateway: Multiple models, frequent prompt changes, explicit cost control. Introduces unified token budget, model routing, prompt canary, model‑level monitoring.

Stage 4 – Multi‑tenant governance & offline recompute: Tags serve many business lines, need versioned snapshots, audit, tenant quotas, batch replay.

Each stage adds capabilities only when the previous stage shows bottlenecks.

11. Capacity risk points for a 100 k QPS target (not measured results)

Only a fraction of traffic should reach the model; use rule‑based shortcuts, deduplication, cache reuse, and budget protection.

Redis hotspots are usually lock keys, latest snapshot keys, and quota counters; mitigate with short TTLs, local + second‑level caches, and bucketed counters.

Scaling Java services does not automatically scale the model side; watch Python gateway loops, model API concurrency limits, token budget, and Kafka backlog.

Cost overruns are often discovered later; enforce real‑time token budgeting, prompt size limits, versioned snapshots, and bounded async retries.

12. AI coding accelerates delivery, not decision‑making

AI helps generate gRPC skeletons, K8s manifests, unit tests, mock and load‑test scripts, and repetitive exception branches, but engineers must still define state machines, idempotency, budget policies and failure responses.

13. Pre‑launch validation checklist

Link correctness: Duplicate requestId should produce a single task; concurrent requests for the same contentId should have only one synchronous executor; sync timeout must trigger async task; async success must converge snapshot and task state.

Capacity validation: Benchmark rule‑out, cache‑hit, lightweight model and heavy model paths; collect P95, P99, error distribution and fallback ratios; monitor Kafka backlog, Redis hot keys, thread‑pool queues and gateway timeout rates.

Cost validation: Measure average input tokens per content length; compare tenant cost across model strategies; evaluate impact of degradation, compression and cache reuse.

Replay validation: Verify prompt upgrades can recompute historical content; ensure dead‑letter tasks can be replayed by time window or tenant; confirm old snapshots remain traceable after model switches.

Scenario validation: Test that rule‑based shortcuts, cache reuse, lightweight model and budget‑driven async paths behave as expected under realistic traffic mixes.

14. When to adopt this architecture and when not to

Suitable scenarios:

Large content volume with tags consumed by multiple downstream systems.

Both real‑time latency and cost are critical.

Tags require versioning, replay and audit capabilities.

Team is willing to maintain governance complexity.

Unsuitable scenarios:

Single low‑traffic business with modest concurrency.

Tags are only used for offline reporting.

Team lacks ability to manage task tables, monitoring, retry semantics.

Tag value has not been validated, making heavy architecture premature.

In low‑concurrency cases, a simple model call plus an async task table often suffices. Upgrade to the full five‑plane design only when tagging becomes core infrastructure.

15. Conclusion – The difficulty lies in containing AI, not in AI itself

Integrating large models is straightforward; the challenge is making them controllable, rollback‑able, auditable and degradable under high concurrency. Answering questions such as which requests must be synchronous, how tags are versioned and audited, how to degrade under timeout or budget exhaustion, and how stale edits are invalidated, turns AI into a true engineering accelerator rather than a source of new uncertainty.