Choosing the Right Elasticsearch Pagination Method: from/size, search_after, Scroll API, and PIT

Elasticsearch offers four common pagination approaches. This guide analyzes each method’s syntax, strengths, weaknesses, and appropriate use cases, helping developers choose the most suitable technique for their specific requirements.

1. Using from and size

The most straightforward pagination method sets the from parameter to indicate the starting offset and size to specify the number of records to return.

GET /index/_search
{
  "from": 10,
  "size": 10,
  "query": {
    "match": { "field": "value" }
  }
}

Advantages

Simple and intuitive : Easy to implement for most basic pagination needs.

Broadly supported : Native support in the Elasticsearch Search API.

Disadvantages

Performance issues : High from values cause Elasticsearch to skip many documents, increasing query time.

Resource consumption : Large offsets consume more memory and CPU, potentially impacting cluster performance.

Applicable Scenarios

Shallow pagination : Ideal for the first few pages (e.g., page 1‑10).

Small datasets : Suitable when the total data volume is modest and pagination requirements are simple.

2. Using search_after

search_after

enables deep pagination by providing the sort values from the previous page, allowing the next page to be fetched efficiently.

GET /index/_search
{
  "size": 10,
  "query": { "match": { "field": "value" } },
  "sort": [
    { "timestamp": "asc" },
    { "_id": "asc" }
  ],
  "search_after": [ "2023-01-01T00:00:00", "some_id" ]
}

Advantages

Efficient deep pagination : Performs better than from/size for large page numbers.

Strong deduplication : When combined with a unique sort field (e.g., _id), it avoids duplicate results.

Disadvantages

State management : Clients must store the previous page’s sort values, adding implementation complexity.

No random page jumps : Only sequential navigation is possible.

Applicable Scenarios

Deep pagination : Suitable for accessing large volumes of data where performance matters.

Continuous data streams : Ideal for log retrieval, real‑time analytics, and similar use cases.

3. Using the Scroll API

The Scroll API is designed for bulk retrieval of massive result sets by keeping a snapshot of the index at query time.

POST /index/_search?scroll=1m
{
  "size": 100,
  "query": { "match_all": {} }
}

# Retrieve subsequent batches
POST /_search/scroll
{
  "scroll": "1m",
  "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAA..."
}

Advantages

Handles large data volumes : Stable performance for exporting or batch‑processing huge datasets.

Avoids page‑jump issues : The snapshot prevents data changes from affecting pagination results.

Disadvantages

Resource consumption : Maintaining the scroll context occupies cluster resources, especially under high concurrency.

Not suited for real‑time search : Intended for one‑off retrieval, not interactive pagination.

Applicable Scenarios

Bulk data export : Data migration, backup, etc.

Large‑scale analysis : Scenarios requiring processing of many documents in a single operation.

4. Using Point in Time (PIT)

PIT provides a time‑based view that remains consistent across multiple pagination requests.

POST /index/_search?pit=true&size=10
{
  "sort": [...],
  "query": { ... }
}

# Subsequent request using pit_id
POST /index/_search
{
  "pit": { "id": "some_pit_id", "keep_alive": "1m" },
  "sort": [...],
  "query": { ... },
  "search_after": [ ... ]
}

Advantages

Consistent view : Guarantees the same data snapshot even if the index changes.

Combines with search_after : Improves efficiency and consistency for deep pagination.

Disadvantages

Increased complexity : Requires managing PIT lifecycles and releasing resources.

Resource consumption : Keeping PIT sessions open consumes cluster resources.

Applicable Scenarios

Consistent pagination : Multi‑user environments where each user must see the same data snapshot.

Deep pagination with consistency : Use together with search_after for efficient, reliable paging.

5. How to Choose the Right Method

5.1 Based on Pagination Depth

Shallow pages : Use from / size for simplicity and acceptable performance.

Deep pages : Prefer search_after or Point in Time to maintain performance and avoid resource waste.

5.2 Based on Data Consistency Requirements

No strict consistency needed : from / size suffices for relatively static data.

Consistent view required : Use Point in Time to ensure pagination sees a stable snapshot.

5.3 Based on Use Cases

Interactive user pagination : Typically from / size for most web applications.

Batch processing or export : Employ the Scroll API for one‑off large data jobs.

5.4 Based on Resource and Performance Considerations

Limited resources : Avoid the Scroll API in high‑concurrency environments.

Performance optimization : For frequent deep pagination, search_after and Point in Time are the preferred choices.

6. Summary

from

+ size: Simple, good for shallow pagination, but performance degrades on deep pages. search_after: Efficient for deep pagination, better performance, but adds complexity and cannot jump pages.

Scroll API: Ideal for bulk data export or processing; not suitable for real‑time interactive pagination.

Point in Time (PIT): Provides a consistent view across pages, best for deep pagination where data consistency matters.

In practice, select the pagination strategy that aligns with your business needs, data volume, pagination depth, and available system resources to achieve optimal performance and user experience.