How Qoder Built Hundred-Billion-Scale Vector Retrieval for AI Coding Agents

Retrieval Is Not "Stuffing More Code" but Finding Task-Relevant Context

Qoder provides an Agentic Coding platform across Desktop, IDE, JetBrains plugins, CLI, and Mobile, with a shared need: letting the agent understand the user's task and project context. For a code modification task, the agent must locate relevant code, call relationships, project conventions, and history. Full codebases cannot fit in model context, so retrieval must first pinpoint task-relevant information and then supplement dependencies. This retrieval spans multi-turn interactions, making per-query latency and cost critical. Access patterns vary: active workspaces need continuous fast context, while low-frequency codebases need low-cost storage with timely response on re-access. Qoder set infrastructure targets: per-codebase monthly cost ≤ ¥1, P95 retrieval latency ≤ 100 ms, support for ten million tenants, balancing hot and cold data access. (These are design targets, not current production measurements.)

One Codebase per Slice: Keeping Global Scale Off Per-Query Scan

Alibaba Cloud Elasticsearch Agent Engine organizes data via Collections and Slices. Qoder maps same-dimension vectors to a Collection, each Codebase to a Slice, each code chunk to a Document. Queries are scoped to the target Slice, touching only the current codebase's data, achieving tenant isolation and preventing global data size from becoming per-query scan burden. Under a reference workload, hot-query P99 ≈ 60 ms with recall ≥ 95%; hundred-million tenants and hundred-billion vectors represent design capacity, not current tenant count. Costs use object storage for bulk data, SSD and memory cache for active access, avoiding long-term high-performance media for cold data. The one-codebase-per-slice design balances multi-tenant isolation, resource sharing, and on-demand access.

End-to-End Retrieval Pipeline: From Query to Fused Context

A user query enters Qoder, undergoes query processing, rewriting, and intent recognition, then multi-channel recall across vectors, keywords, symbols, commit records, and Wiki. Results are merged and deduplicated, followed by subgraph construction to expand call and dependency relationships, then ranking and context fusion. Xia noted: "ES Agent Engine handles the vector recall capability; the full retrieval chain also includes upstream understanding, expansion, and downstream result processing."

Production Engineering: From Request Success to Business Success

Index building starts with upload: Manager receives tasks, schedules Workers via message queue; Workers call Embedding Service to generate vectors, then write to KV Cache and Elasticsearch. Business layer provides unified write, query, and lifecycle interfaces, persisting each Codebase's actual backend routing for consistent read/write targets.

Production details: Collections and Slices created on first non-empty write; Bulk requests split by payload size with per-item result checks because HTTP success ≠ per-document success; queries must specify Slice and filters to limit recall scope; underlying scores normalized to align with upper-layer threshold semantics.

Security and reliability: Elasticsearch stores only vectors and necessary metadata (e.g., location), not raw code or chunk content. Query side uses multi-node and replica configuration, client separates read/write addresses, retries transient errors. Operations trace Codebase through entry, actual backend, HTTP request, Bulk per-item results, and latency. "Production reliability acceptance is not 'API returns success' but business read/write success, data integrity, and recoverability after failure."

Hundred-Billion-Scale Production Metrics: Performance, Throughput, Cost Under Defined Methodology

Current storage reaches hundred-billion documents: 30+ ECS for writes, 5+ ECS for queries; peak query QPS 1000, agent concurrent retrieval QPS >20, write peak >1 GB/s. Assuming 10k vectors per codebase, shared cluster, and specific discounted pricing, estimated per-Codebase monthly infrastructure cost ≈ ¥0.016. This is a specific-assumption amortized estimate, not a fixed price or full business cost.

In Qoder's test methodology, Elasticsearch pure vector recall P50/P95 = 11.73 ms / 14 ms vs. comparable product 30.47 ms / 36.25 ms (P95 reduction ~61.4%); write bandwidth 2 GB/s vs. 400 MB/s (~5×); total cost ~15% of comparable product baseline. These latency, write, and cost figures use different test/calculation conditions and should not be combined or extrapolated to all loads; actual performance varies with data scale, resource config, concurrency, and cache state.

Toward Unified Retrieval Across Code, Memory, and Enterprise Knowledge

Code retrieval is one part of agent context engineering. Qoder plans to extend the same engineering capabilities to user/project memory, enterprise knowledge bases, and cross-repository knowledge. Memory requires isolation, timeliness, and deletion mechanisms; enterprise knowledge bases need permission filtering during mixed document/Wiki retrieval while preserving source and version; cross-repo knowledge must link code, issues, and review records for traceable context across projects.

Future agents would first identify task goals and authorization scope, then select code, memory, or knowledge-base retrievers, unifying deduplication and reranking across sources. "Retaining the most helpful content within a limited token budget while preserving evidence sources is the core direction for fused retrieval evolution." Qoder will continue exploring fused retrieval of code, memory, and enterprise knowledge to give agents more relevant, well-sourced, and explainable context.

This article is compiled from the 2026 Yunqi Conference AI Search sub-forum presentation. All scale, performance, throughput, and cost data reflect the versions, configurations, test, and calculation methodologies disclosed in the talk; future scenarios are planning directions and do not indicate all capabilities are live.

Alibaba Cloud Elasticsearch Agent Engine: https://help.aliyun.com/zh/es/user-guide/what-is-ai-engine-edition?spm=a2c4g.11186623.0.0.601f6489qQItFo