Dayu AI Model Gateway: More Than a Reverse Proxy – Design & Practice

Why a Model Gateway Is More Than a Reverse Proxy

A simple reverse proxy that unifies protocols, swaps keys, and adds rate limiting works for a few models but breaks down at scale. Streaming conversion must understand event relationships; stateful requests cannot randomly switch channels; unbounded retries worsen overload; status codes and billing can diverge. Therefore the gateway must comprehend the content it forwards. The architecture splits into three planes by latency and consistency requirements:

Control Plane: Manages providers, model routing, key groups, and quotas. Configuration and tickets are persisted with strong consistency and auditability.

Data Plane: Handles online request authentication, protocol adaptation, rate limiting, and routing. Configuration snapshots with version numbers are synced to gateways; nodes only accept newer versions, preventing stale configs from overwriting fresh ones.

Accounting Plane: Data plane emits standardized usage events asynchronously at request end; accounting service matches prices and computes costs.

This separation ensures management jitter never reaches the call path and billing never slows online requests.

Streaming Protocol Conversion: Don't Lose Data in Translation

Unified Intermediate Representation

The gateway simultaneously supports OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini. Writing pairwise converters would require dozens of implementations. Instead, each protocol implements only two steps: decode its events into a unified intermediate representation, then encode that representation back into its own format. When ingress and upstream protocols match and no fields need rewriting, responses are passed through verbatim with only side‑channel usage parsing; otherwise the decode–intermediate–encode path is used.

Anthropic Messages Streaming Example

message_start           usage.input_tokens=1006 cache_read_input_tokens=45696
content_block_start     index=0 type=text
content_block_delta     index=0 "Let me search…"
content_block_start     index=1 type=tool_use id=toolu_01 name=search
content_block_delta     index=1 input_json_delta {"query": "Large model
content_block_delta     index=1 input_json_delta gateway"}
message_delta           stop_reason=tool_use usage.output_tokens=86
message_stop

Three Critical Conversion Points

Token Accounting: Anthropic's input_tokens excludes cache hits (recorded separately in cache_read_input_tokens), while OpenAI's prompt_tokens includes them. The decoder sums input, cache read, and cache write for total input; cache details are preserved. When encoding back to Anthropic, the cache portion is subtracted; negative results trigger an error rather than producing a wrong bill. In the example, conversion to OpenAI yields prompt_tokens=46702, cached_tokens=45696.

Usage Aggregation Timing: Input, output, and cache usage are scattered across stream events. The decoder accumulates per request event‑by‑event, merges into a complete standardized usage record, then hands it to metering and the encoder for downstream delivery per target protocol.

Tool Call Indexing and Identification: Tool call id and name appear only in the start event; subsequent parameter deltas carry only an index. The converter must remember the index‑to‑call mapping. Protocols differ: Responses uses output item ID and position; decoding normalizes to zero‑based tool indices. Encoding to Anthropic renumbers content blocks sequentially, assembles parallel tool parameters fully within each block before emission to avoid interleaving. Call IDs are standardized to tool_ prefix; the same original ID always maps to the same result so the next round's tool result matches.

Stateful Stream Converters

Stream decoders and encoders are created per request, each holding its own state. Text deltas are forwarded immediately without adding first‑token latency; cross‑event data like usage and tool call mappings accumulate until stream end. Unknown upstream events (e.g., web search results) are passed through unchanged when protocols match; if the target protocol cannot express them, an explicit error is raised rather than silently dropping data. Standardized usage includes input, output, cache read, cache write, and reasoning tokens, serving as the single source of truth for monitoring and billing. Regression tests cover all protocol pair combinations; real‑world calls and reported usage match item‑by‑item.

Requests That Must Not Switch Channels

Responses Session Continuation: Context resides on the provider side; continuation requests must return to the original provider. The gateway retains response‑ID‑to‑provider affinity and routes continuations accordingly, bypassing load balancing. If affinity is missing, the session ID is preserved and passed to the provider for native handling.

Embedding Cross‑Channel Switching Prevention: Different models/dimensions live in different vector spaces; mixing them silently degrades retrieval quality and is hard to debug. When a backup channel cannot confirm consistency with the primary, the gateway blocks the switch — returning an error is far preferable to corrupting the index.

After protocol unification, model switching cost drops significantly. Developers can browse a model plaza, view capabilities/pricing/status, and validate long‑context, structured output, and latency with their own prompts and samples for preliminary selection.

Scheduling: Preventing Fault Amplification

Why Retries Amplify Failures

Assume a channel is overloaded: 100 QPS inbound, most fail. With up to 2 retries per request, the provider sees nearly 300 attempts. Overload is the root cause; unconstrained retries slow recovery and spill pressure to other channels in the same pool.

Unified Scheduling Loop

Load selection, retry, and capacity control run in a single loop:

Load Selection: Chooses the next target — supports weighted distribution, configured order, least‑recently‑used, and recent latency awareness.

Retry Policy: Only for network errors and configured status codes; per‑target attempt limit; on exhaustion, try another target.

Capacity Accounting: Every upstream attempt consumes the provider's request quota but does not double‑count against the caller's quota. The caller isn't charged extra for gateway retries; the provider isn't overwhelmed by retries; backup channels also respect remaining capacity.

Backoff Strategy: Faulty channels are deprioritized, not ejected. If all targets are in backoff, the gateway still attempts enabled ones, avoiding total blackout.

Streaming Request Commit Point

Streaming requests have a natural boundary: before the first valid content reaches the caller, the gateway can transparently switch channels — the caller notices nothing. After content starts flowing, switching would yield a disjoint response. The gateway completes fault tolerance before the commit point; the caller can then design appropriate interaction for post‑commit interruptions.

Shared Capacity Borrowing

Rate limiting constrains both callers and providers. Key groups limit per‑caller QPS; experience‑type traffic has a shared pool that can borrow idle capacity when allowed. Borrowing stops when provider load rises; idle capacity isn't wasted, and online traffic gets priority under contention.

Observability: Success Doesn't Mean No Problems

The opening example returned 200 then disconnected mid‑stream — a success‑only monitor would mark it healthy. Another request failed on two channels, succeeded on the third — two channel faults hidden by final success. Therefore the gateway records four distinct categories per request: callers see final result; operators see attempt details.

HTTP Status: Status code returned to caller.

Model Final Result: Normal completion, truncated, cancelled, or mid‑stream failure.

Upstream Attempt Details: Which channel each attempt hit and its outcome.

Rate Limit Events: Which layer and dimension triggered limiting.

Latency Breakdown

"Model is slow" is the top complaint, but root causes vary widely. The gateway splits latency into distinct signals:

Slow First Token: Metric — time to first valid content. Causes — queueing, long input, slow upstream processing.

Normal First Token, Slow Output: Metric — output speed. Cause — insufficient upstream generation capacity.

Segments Normal, High Total Latency: Metric — overall latency, output length. Cause — excessive output length; optimize prompt.

Queued or Rejected: Metric — active concurrency, rate limit events. Cause — quota or channel capacity shortage.

With metrics separated, business, platform, and ops troubleshoot from the same data. Each call record links application, key group, key, owner, routed model, and actual provider model (credentials masked). Owners drill from group usage to specific keys/models; shared credentials still attribute traffic correctly. Quota alerts operate at group, shared pool, and exclusive key levels with thresholds and intervals to avoid notification fatigue.

Billing: Every Cost Reproducible

Cost Estimation Before Integration

Cost depends on input/output length and call volume, so estimation happens at selection and approval. Businesses enter average input/output tokens and call volume in the model plaza and application; the platform combines model pricing to estimate per‑call, daily, and monthly costs; batch tasks get total cost estimates. Applicants and approvers discuss model choice and budget against the same numbers.

One Billing Model, Multiple Rules

Billing rules differ widely across models:

Standard Tokens: Input, output, cache, reasoning sub‑items.

Tiered Tokens: Price rule selected by configured thresholds.

Time‑of‑Day Tokens: Pricing by time windows, weekday/weekend rules.

Vector Input: Embedding model input and granular usage.

Image Tokens: Text and image input/output sub‑items.

All rules apply to the standardized usage from Section 2, with discounts layered on. New billing types only extend rules, not the protocol layer. Models without price config are explicitly marked "unaccounted" — never recorded as zero — to avoid being mistaken for free.

Price Locked at Request Entry

Prices change, but historical costs must be reproducible. The platform determines and saves the pricing basis the moment a request enters the gateway, creating an immutable price snapshot.

Cross Price Change: New price effective 10:00; a request entering at 09:59 and ending at 10:02 is billed at the old price.

Cross Time Boundary: Request entering at 23:59:58 keeps its original time‑window pricing even if it ends in the next window.

Model Decommissioned: After config deletion, historical requests' pricing basis remains traceable.

During bill reconciliation, every charge shows which price version and which usage sub‑items were used.

Separate Quota Check and Cost Calculation

LLM costs are only known after generation ends. Pre‑deducting then refunding/adjusting causes two problems: output length is notoriously hard to predict, and every request would add a synchronous dependency on the accounting system. The gateway splits them: a lightweight quota check at entry (no accounting system dependency, so accounting hiccups don't affect online calls); actual cost is computed asynchronously after request completion using real usage — each deduction matches a real call, no refunds needed. Combined with quota alerts and usage monitoring, businesses see consumption trends in time.

Resources and Processes: Managing Both Money and Permissions

Amount and Quota Are Two Dimensions

Amount quota caps spend; model quota (RPM/TPM) caps call speed. Adding budget doesn't speed up calls; raising RPM/TPM doesn't grant budget. They are requested and managed separately. When adding a backup model, if total spend growth is limited, only quota needs increase — no extra budget request per backup.

Online services use monthly quota key groups (amount resets calendar month); tests/backfills use cumulative quota key groups (authorized and released per task cycle).

Key Groups and API Keys

Key groups own business resources; API keys are just call credentials. One business can have multiple keys; adding callers or revoking keys doesn't require redefining resources. Keys can draw from group pool or receive exclusive quota. Exclusive allocations obey constraints:

Exclusive allocation deducted from group quota; sum of allocations ≤ group quota.

Quota adjustments cannot go below already consumed amount.

Key authorization cannot exceed group scope.

Disabling a key does not auto‑release its exclusive quota; re‑enabling retains it.

Allocation released only at task expiry; key and history preserved for audit.

Idle Resource Reclamation

Low‑usage resources aren't necessarily useless — backup models sit idle but are fault‑tolerance fallbacks. Before reclamation, owners are notified, observed for a period, and manually reviewed. Cumulative tasks have a startup buffer beyond approved period; expiry reclaims only the current authorization's reclaimable quota and unused amount; other valid authorizations in the same group are unaffected. Model deprecation follows similar flow: platform identifies all consumers, notifies owners, retains records; businesses validate replacements; final shutdown requires admin confirmation of migration completion.

Consistency Between Approval and Execution

Resource requests flow through company OA approval; on approval the gateway executes grants. Any interruption among approval system, gateway config, or background jobs can leave intermediate states (approved but not executed, half‑executed, duplicate execution) — a distributed consistency problem. Handled via:

Fixed Approval Content: Approved content stored as immutable version at approval time; execution only recognizes that version; later application edits don't affect it.

Atomic Authorization Commit: Permissions, RPM/TPM, and amount committed in a single transaction — all succeed or all roll back; no partial state like amount added but permissions missing.

Recoverable, Idempotent Execution: Background tasks drive execution; closing the page doesn't stop it. On recovery, task verifies execution identity, stage, and existing results. Expired tasks cannot submit; already‑executed steps return receipts, preventing duplicate resource grants.

Platform retains notification records, ticket history, and audit logs to verify message delivery, process completion, and config changes. If granted quota differs from expectation, owners check ticket for approved details, then execution receipt, then subsequent modifications — no reliance on memory. Other platforms can reuse gateway's authorization, metering, and cost query via open APIs using separate management AK/SK (distinct from model API keys), scoped by interface permissions and bound key groups.

Next Steps

Model Trial Applications: Separate trial from production; trial resources have clear purpose; successful validation converts directly to formal integration.

Model Evaluation: Combine business eval sets, machine eval, and human eval with gateway's latency, usage, and cost data to compare effectiveness, speed, and cost together.

Tool Capability Reuse: Distribute authorized reusable business tools, letting models invoke existing capabilities within explicit permission boundaries.

Batch Job Scheduling: Schedule deferrable tasks (tests, backfills) using idle capacity and task deadlines while protecting online traffic.

Conclusion

Dayu AI Model Gateway consolidates protocol adaptation, scheduling fault tolerance, observability, and metering/billing into one layer. Businesses integrate once to get unified call semantics, controlled retries, and reproducible costs. As upstream channels, prices, and protocols continue to evolve, the gateway will keep absorbing these changes so businesses can focus on their own scenarios.