Why Deep Pagination Breaks Elasticsearch and How to Fix It

In the distributed search domain, Elasticsearch (ES) is praised for its performance and scalability, but a seemingly harmless operation—deep pagination (using from / size)—often becomes a hidden performance killer. When users request page 1000 or even 10000, the ES cluster’s CPU and memory can spike, response times degrade sharply, and the cluster may appear dead.

1. Scatter‑Gather Mechanism Under Deep Pagination

Elasticsearch stores data across multiple shards. A query typically goes through two phases:

Scatter Phase : The coordinating node broadcasts the query to all relevant shards. For a request like from: 10000, size: 10, each shard must find matching documents, sort them, and return the top 10000 + 10 = 10010 records to the coordinator.

Gather Phase : The coordinator receives N * 10010 records (where N is the number of shards), loads all of them into memory, performs a global sort, discards the first 10000, and finally returns the last 10 to the client.

This process is costly because:

Data Transfer Overhead : Data transferred between shards and the coordinator grows linearly with pagination depth.

Memory Pressure : The coordinator must allocate enough memory to hold all returned records, easily causing OOM.

CPU Consumption : Sorting N * (from + size) records is CPU‑intensive.

Thus, the client receives only 10 records while ES processes tens of thousands, discarding 99.99% of the work.

2. Solutions: Ditch from/size

Because from/size is inefficient for deep pagination, Elasticsearch offers two powerful alternatives:

Solution 1: scroll API – Built for Massive Data Export

The scroll API works like a database cursor. The first scroll request creates a snapshot of all query results and returns a _scroll_id (a bookmark). Subsequent requests use this _scroll_id to fetch the next batch without re‑executing the query or sorting.

Advantages : Extremely high performance, minimal memory pressure on the coordinating node, ideal for bulk export, data migration, or offline analysis.

Disadvantages : Operates on a historical snapshot, so it cannot reflect real‑time changes; only supports forward pagination; the snapshot consumes server resources and must be cleared after use.

Solution 2: search_after – High‑Performance Real‑Time Pagination

The search_after API provides a stateless, forward‑only pagination method. It uses the sort values of the last document from the previous page as a “bookmark” to locate the next page’s start. A unique field (e.g., _id) is often added to ensure deterministic ordering.

Advantages : No server‑side state (no scroll_id), excellent scalability, each request is lightweight and returns real‑time data.

Disadvantages : Only supports forward pagination; cannot jump directly to an arbitrary page number.

3. Practical Mixed Pagination Strategy

Combine the strengths of both approaches by setting a depth threshold (e.g., from + size <= 10000, matching ES’s index.max_result_window) and dynamically switching pagination methods:

For shallow requests (within the threshold), use the traditional from/size to allow free page jumps.

For deep requests (exceeding the threshold), hide page‑number input in the UI and switch the backend to search_after for efficient forward scrolling.

This hybrid approach lets you enjoy the convenience of from/size for most cases while gracefully falling back to search_after when performance limits are reached.

4. Conclusion: Smart Pagination Choices

In summary: from + size is the default UI pagination method; keep it within a safe shallow range. search_after is the go‑to solution for real‑time infinite scroll and deep data exploration. scroll is a specialized tool for bulk data export and should not be used for regular online pagination.

Adopting a dynamic switch between from/size and search_after provides the best balance between user experience and system performance in complex business scenarios.