How Elasticsearch’s Doc Values Boost Sorting and Aggregation Performance

Elasticsearch’s internal data structures and storage mechanisms are essential for efficient search and fast query responses, with column‑store Doc Values playing a core role.

1. What are Doc Values

Doc Values are an internal Elasticsearch data structure that stores the data needed for sorting and aggregation at the field level. Unlike traditional row storage, where each document stores its field values together, Doc Values use a columnar layout, organizing data by field rather than by document, which optimizes read performance for sorting, aggregation, or script calculations.

2. Why Doc Values are Needed

Sorting and aggregation are critical for processing large data sets, but the inverted index—while excellent for full‑text search—does not efficiently support these operations because it is designed to locate documents containing specific terms, not to collect all terms from a set of documents.

When a field is sorted or aggregated, Elasticsearch must visit every matching document to retrieve that field’s value. For example, consider a term‑document matrix:

Term      | Doc_1 | Doc_2 | Doc_3
-------------------------------
 brown    |   X   |   X   |
 dog      |   X   |       |   X

Finding documents containing “brown” is fast, but aggregating all unique terms from those documents requires scanning every term in the index, which becomes slower as the number of terms and documents grows.

Doc Values solve this by storing a complete, sorted list of field values per document, mapping documents to their terms rather than terms to documents, making field‑value collection highly efficient.

3. How Doc Values Work

When a document is indexed, Elasticsearch creates Doc Values for fields that need sorting or aggregation. These values are compressed, column‑oriented, and stored separately from the inverted index, optimized for fast random access.

Data Generation & Storage : For each field, Doc Values are generated at index time. Numeric, date, or enum fields are stored directly, while text fields may store one or more token results.

Memory & Disk Usage : Doc Values are serialized to disk, reducing JVM heap pressure. Because they are column‑oriented, they can be efficiently cached by the OS file system cache, allowing reads from memory instead of disk during sorting or aggregation.

Query Process : During a sort or aggregation query, Elasticsearch directly accesses the field values from Doc Values without traversing the inverted index, benefiting from the pre‑sorted, per‑document layout.

Performance Optimisation : Accessing Doc Values is typically much faster than gathering values from the inverted index, which is tuned for document lookup, not field‑value collection. Doc Values also support certain filters, such as geo‑filters, that require rapid field access.

Relation to Inverted Index : Doc Values complement, not replace, the inverted index. The inverted index remains responsible for full‑text search.

4. Types and Storage of Doc Values

Elasticsearch supports various Doc Values types, including numeric, date, IP address, and binary, each with a specific encoding to optimise space and query speed. For example, numeric Doc Values may use efficient compression algorithms, while dates are stored as comparable long timestamps.

4.1 Persistence

Doc Values are generated per segment and are immutable; once created they never change. They are serialized and persisted to disk, allowing the operating system to cache them outside the JVM heap. When the working set fits in available memory, the OS loads Doc Values into RAM for very fast reads. If the working set exceeds memory, the OS swaps portions to disk, preventing heap‑out‑of‑memory crashes, though access speed may degrade.

4.2 Compression

The columnar layout lends itself to compression, especially for numeric fields. Compression strategies include:

If all values are distinct (or missing), a marker records them.

If values are < 256, a simple encoding table is used.

If values share a greatest common divisor (e.g., all are multiples of 100), the divisor is used to shrink the numbers.

Otherwise, an offset from the smallest value is encoded.

String fields are first encoded to numeric identifiers via a lookup table, then compressed like numeric types.

5. Using Doc Values

Doc Values are enabled by default for all non‑analyzed fields because analyzed text fields generate many tokens, reducing the benefit. To save disk space and speed up indexing, you can disable Doc Values on a field that does not require sorting, aggregation, or scripting:

PUT my_index
{
  "mappings":{
    "my_type":{
      "properties":{
        "session_id":{
          "type":"string",
          "index":"not_analyzed",
          "doc_values":false
        }
      }
    }
  }
}

Disabling Doc Values removes support for aggregation, sorting, and script access on that field.

You can also configure a field to have Doc Values but no inverted index, enabling aggregation while preventing term‑based search:

PUT my_index
{
  "mappings":{
    "my_type":{
      "properties":{
        "customer_token":{
          "type":"string",
          "index":"not_analyzed",
          "doc_values":true,
          "index":"no"
        }
      }
    }
  }
}

This setup supports aggregation but not full‑text queries because no inverted index is created.

Conclusion

Doc Values are a key component of Elasticsearch performance optimisation. By pre‑computing and storing field values in a column‑store format, they dramatically improve the speed of sorting, aggregation, and certain filter operations, making them essential for clusters that handle large data volumes and complex queries.