Alibaba Cloud Elasticsearch Agent Engine: 70% Cost Reduction, 60ms P99 for 100M Tenants

Agent-Era Search Infrastructure Challenges

At the 2026 Yunqi Conference AI Search Forum, Alibaba Cloud senior technical expert Yang Kongshi presented the Elasticsearch Agent Engine Edition, designed for 100 million tenants and 100 billion vectors. Enterprise knowledge bases, code workspaces, industry repositories, and personal memories are becoming external contexts repeatedly invoked by agents. This creates a new contradiction: the number of independent retrieval spaces grows, data must be retained long-term, but only a fraction is active at any moment.

"Data needs long-term storage, but resources cannot stand by for every piece of data."

Traditional Elasticsearch deployments face massive shards when indexing per tenant, while shared indexes require extra tenant filtering. Vector indexes and multi-replica SSDs occupy resources long-term, and storage-compute coupling means scaling and failure recovery require data migration. As total data grows, a single query touching many other tenants' data increases useless reads and computation.

User goals: retrieval P95 < 100 ms, recall ≥ 95%. Existing solutions show latency > 1 second (percentile unspecified).

Storage-Compute Separation: OSS for Persistence, Cache for Active Working Set

The Agent Engine Edition persists index data, write-ahead logs (WAL), and metadata uniformly to OSS. The write layer (index nodes) builds indexes; the query layer (query nodes) serves searches. Both layers can be configured independently. Node-local memory and SSD serve primarily as cache and temporary space; scaling or replacing nodes does not require moving the full dataset.

Yang likens this to a "desk and library": the desk holds only currently used materials, long-term materials stay in the library. Active tenants' data is loaded into cache on demand; cold data remains in OSS. Businesses can pre-warm target workspaces to reduce first-access latency.

"Making resource investment depend on active load rather than full data volume is key to lowering long-tail tenant resident overhead."

The control plane monitors CPU, memory, disk, and I/O watermarks for write and query layers separately. Automatic compute scaling is planned for October 2026, adding nodes at high watermarks and reclaiming at low watermarks.

Slice: Tenant Boundary as First-Class Organizing Unit

In data organization, one Slice corresponds to one tenant's retrieval space. A Slice Collection aggregates many Slices under a single logical entry point; backend physical indexes expand on demand. Businesses need not modify entry points when underlying indexes change; queries, caching, and pre-warming execute around explicit tenant boundaries.

DiskBBQ Vector Index: Hierarchical K-Means + 1-Bit Quantization

For vector retrieval, DiskBBQ first clusters similar vectors via hierarchical K-Means, then quantizes vectors to 1-bit codes, writing them to disk in cluster order to form sequentially readable blocks. The 1-bit code width is 1/32 of 32-bit float, but this does not equal 32× overall storage compression. Because no graph construction is needed, DiskBBQ builds much faster: on a 1-million-vector benchmark, DiskBBQ took 94 seconds vs. HNSW's 1,054 seconds — roughly 10× speedup.

In multi-tenant scenarios, Sliced DiskBBQ pushes tenant boundaries into the vector index: queries locate the current tenant's data range directly, incurring zero I/O and zero scoring for other tenants' data.

"A single knowledge base's query should not bear increasing irrelevant reads and computation because other knowledge bases grow."

Cost and Query Performance Under Reference Load

Using self-built Elasticsearch cost as 100%, under the same reference load (10 billion documents, 10 thousand knowledge bases), Agent Engine Edition overall cost is ~30% — a ~70% reduction. Breakdown: storage ~20%, memory ~25%, CPU ~50%. Savings come mainly from storage-compute separation and on-demand caching; low-frequency tenants no longer consume resident resources.

Query experience: hot query P99 ~60 ms, vector recall >95%; self-built Elasticsearch latency ~2 seconds, recall ~70%. Cold queries reading from OSS take ~600 ms; pre-warming reduces first-query wait.

Product Capabilities and Use Cases

The Alibaba Cloud Elasticsearch console now offers an Agent Engine Edition entry. Users can enable Slice Collection for massive multi-tenant retrieval and configure index/query nodes separately. Three capabilities around tenant data spaces:

Warm Slice (preheat) : load data ahead of user opening a workspace or starting a session.

Copy Slice : create independent environments.

Branch Slice : copy-on-write, creates lightweight branches in seconds; currently in Beta.

Applicable scenarios: enterprise knowledge bases & RAG, AI Coding, legal/finance AI, agent memory, mobile and in-vehicle assistant memory. For AI Coding, each code repository or workspace is an independent retrieval space. Agents execute tasks by cross-file retrieval of source code, tests, and comments, combining vector, keyword, and filter conditions. Single dataset requirements: 100,000+ files, full index P95 ≤ 1 minute, update interval ≤ 10 seconds.

Conclusion

Yang summarized that the core of agent-oriented search infrastructure is letting "query scope be determined by the tenant's own data." As agent count, knowledge spaces, and long-term memory grow continuously, only by decoupling persistent data from active compute and organizing data and queries by tenant boundaries can search infrastructure balance scale, experience, cost, and operational efficiency.

Note: 100M tenants and 100B vectors are architectural design capacities. Performance and cost data reflect a reference load of 10B documents and 10K knowledge bases; actual results vary with dataset, resources, cache, and retrieval parameters. Cost savings vary with active data ratio, resource configuration, and access load. DiskBBQ build speedup varies with dataset, parameters, and resources. Branch creation latency depends on metadata scale and system load.