From Code Context to 100B Docs: Qoder's Production Vector Retrieval with Elasticsearch

From Code Context to 100B Documents: Production-Grade Vector Retrieval for Agentic Coding

At the 2026 Yunqi Conference AI Search Forum, Alibaba ATH Qoder technical expert Xia Xiaowen presented how Qoder leverages Alibaba Cloud Elasticsearch Agent Engine Edition to build a production-grade vector retrieval foundation for code context. The talk covers architecture choices, engineering practices, and operational results for large-scale code retrieval, and outlines a roadmap toward unified retrieval across code, memory, and enterprise knowledge.

01 Retrieval Is Not "Stuffing More Code" but Finding the Context Needed to Complete the Task

Qoder provides an Agentic Coding platform spanning Desktop, IDE, JetBrains plugins, CLI, and Mobile, with Agent SDK and Cloud Agents exposing agent capabilities. All entry points share a common need: the agent must understand the user's current task and the project context behind it. Taking a feature modification as an example, the agent needs not only the relevant code locations but also call graphs, project conventions, and historical changes. Because the full codebase cannot fit into the model context, the retrieval system must first locate task-relevant information and then supplement necessary dependencies and background. This retrieval runs through every turn of the development interaction, so single-query latency and cost directly affect overall experience and service cost. Meanwhile, access frequency varies sharply across codebases: active workspaces need continuous, fast context access, while low-frequency codebases require low-cost storage with timely response on re-access. Qoder therefore set infrastructure targets including under 1 yuan per codebase per month, P95 retrieval latency under 100 ms, support for 10M+ tenants, and balanced handling of hot and cold data. (These figures are design targets for architecture selection, not current measured production scale.)

02 One Codebase per Slice: Keeping Global Scale from Becoming a Per-Query Burden

For massive multi-tenant scenarios, Alibaba Cloud Elasticsearch Agent Engine Edition organizes business data via Collections and Slices. In Qoder's mapping, same-dimension vectors go into the same Collection; one Codebase maps to one Slice; one code Chunk maps to one Document. Queries are scoped to the target Slice, touching only the current codebase's data. This achieves tenant isolation while preventing the global data volume from turning into per-query scan overhead. The speaker shared that under a specific reference workload, hot-query P99 is around 60 ms with recall no lower than 95%; 100M tenants and 100B vectors represent product design capacity, not Qoder's current tenant count. On cost, object storage holds large-volume data, while SSD and memory cache serve active access, so cold data does not permanently occupy high-performance media. The one-codebase-per-slice design is the key balance point among multi-tenant isolation, resource sharing, and on-demand access.

03 End-to-End Retrieval Pipeline: From Query Rewriting to Context Fusion

A user query entering Qoder first undergoes query processing, rewriting, and intent recognition, then proceeds through multi-channel recall across vectors, keywords, symbols, commit records, and Wiki. After merging and deduplication, the system builds subgraphs, expands call and dependency relationships, and finally performs ranking and context fusion. Xia noted: "ES Agent Engine Edition handles the vector recall capability; the complete retrieval chain also includes the preceding understanding and expansion and the subsequent result processing."

04 From Request Success to Business Success: Production Engineering Across Every Stage

Index building starts at upload: the Manager receives tasks and schedules Workers via a message queue; Workers call the Embedding Service to generate vectors, then write to both KV Cache and Elasticsearch. The business side exposes a unified interface for write, query, and lifecycle operations, while persisting each Codebase's actual backend routing to ensure subsequent reads and writes hit consistent storage targets.

In production, the system must handle many easily overlooked details. Collections and Slices are created lazily on first non-empty write. Bulk requests are batched by payload size, and each item's execution result is checked individually because HTTP success does not guarantee every write succeeded. Queries must explicitly specify the Slice and filter conditions to bound the recall scope. Underlying scores undergo normalization to keep semantics consistent with upper-layer thresholds.

Security and reliability run through the full chain. Qoder stores only vectors and necessary metadata (e.g., location) in Elasticsearch, not raw code or chunk content. The query side uses multi-node and replica configurations; the client separates read and write addresses and retries transient errors. Operations trace the entry point, actual backend, HTTP request, bulk per-item result, and latency. The acceptance criterion for production reliability is not "API returns success" but business read/write success, data integrity, and recoverability after failure.

05 100B Documents in Production: Performance, Throughput, and Cost Measured with Clear Definitions

According to the talk, Qoder's current storage scale has reached 100B+ documents, with 30+ ECS instances for writes and 5+ ECS instances for queries. Peak query QPS hits 1000, agent concurrent retrieval QPS exceeds 20, and write peak surpasses 1 GB/s. Assuming 10K vectors per codebase, a shared cluster, and specified discounted pricing, the estimated monthly infrastructure cost per codebase is about 0.016 yuan. This number is an amortized estimate under specific assumptions, not a fixed price, and does not represent full business cost.

In the Qoder test methodology shown, Elasticsearch pure vector recall stage P50 and P95 are 11.73 ms and 14 ms respectively, versus a comparable product's 30.47 ms and 36.25 ms — a P95 reduction of roughly 61.4%. Write bandwidth shows 2 GB/s versus 400 MB/s, about 5x. At 100% for the comparable product's total cost, Elasticsearch corresponds to approximately 15%. The speaker cautioned that these latency, write, and cost figures come from different test and calculation conditions and should not be combined or extrapolated as a single result for all loads; actual performance varies with data scale, resource configuration, concurrency, and cache state.

06 Toward Unified Retrieval Across Code, Memory, and Enterprise Knowledge

Code retrieval is only one part of agent context engineering. Qoder plans to extend the same engineering capabilities to user and project memory, enterprise knowledge bases, and cross-repository knowledge. Memory requires attention to isolation, timeliness, and deletion mechanisms. Enterprise knowledge bases need permission filtering during mixed document and Wiki retrieval while preserving source and version. Cross-repo knowledge must link code, issues, and review records to provide traceable context for cross-project tasks.

In the future, the agent would first identify the task goal and authorization scope, then select among code, memory, or knowledge-base retrievers, and finally unify deduplication and reranking across sources. The core direction for fused retrieval is retaining the most helpful content within the limited token budget while preserving evidence sources for explainability. Qoder will continue exploring unified retrieval across code, memory, and enterprise knowledge so that agents obtain more relevant, clearly sourced, and more explainable context.

Disclaimer: This article is compiled from the 2026 Yunqi Conference AI Search Forum presentation materials. All scale, performance, throughput, and cost data reflect the versions, configurations, test, and calculation methodologies disclosed in the talk; future scenarios represent planned directions and do not indicate that all capabilities are already launched.

Alibaba Cloud Elasticsearch Agent Engine Edition: https://help.aliyun.com/zh/es/user-guide/what-is-ai-engine-edition?spm=a2c4g.11186623.0.0.601f6489qQItFo

Alibaba Cloud ES Free Trial: https://free.aliyun.com/?spm=5176.14058969.J_3759233040.1.1c5069afFW3p4q&productCode=elasticsearch