Understanding Elasticsearch Document Scoring and Aggregation Techniques

Elasticsearch (and its underlying Lucene engine) uses a Boolean model to match documents, applying a practical scoring function that incorporates term frequency (TF), inverse document frequency (IDF), and field length normalization (norm) along with modern features such as coordination factors and query term boosts.

Boolean Model – Queries are built using AND , OR , and NOT operators. For example:

full AND text AND search AND (elasticsearch OR lucene)

This query returns documents containing the terms full , text , search , and either elasticsearch or lucene .

Term Frequency (TF) – The weight of a term increases with its frequency in a document. The TF is calculated as the square root of the term count:

tf(t in d) = √frequency

Disabling TF for a field can be done by setting index_options to docs in the mapping:

PUT /my_index
{
  "mappings": {
    "doc": {
      "properties": {
        "text": {
          "type": "string",
          "index_options": "docs"
        }
      }
    }
  }
}

Inverse Document Frequency (IDF) – Rare terms receive higher weight. The IDF formula is:

idf(t) = 1 + log(numDocs / (docFreq + 1))

Field Length Normalization (norm) – Shorter fields receive higher weight. The normalization factor is the inverse square root of the number of terms:

norm(d) = 1 / √numTerms

These three factors are combined during indexing and later used to compute the final score for a term query:

weight(text:fox in 0) [PerFieldSimilarity]: 0.15342641
result of:
  fieldWeight in 0                     0.15342641
  product of:
    tf(freq=1.0), with freq of 1:        1.0
    idf(docFreq=1, maxDocs=1):          0.30685282
    fieldNorm(doc=0):                    0.5

Vector Space Model – Documents and queries are represented as vectors, allowing multi‑term similarity calculations. Example vectors:

[1,2,5,22,3,8]

Term weights can be manually boosted (e.g., happy weight 2, hippopotamus weight 5) and combined into query vectors to compare against document vectors using angular distance.

Scoring Formula – The complete score for a query q and document d is:

score(q,d) = queryNorm(q) · coord(q,d) · Σ[ tf(t in d) · idf(t)² · t.getBoost() · norm(t,d) ] (t in q)

where queryNorm is a global normalization factor, coord rewards documents that match more query terms, and the sum iterates over each query term.

Boosting – Individual query terms can be given higher importance using the boost parameter, influencing the final ranking.

Aggregation Techniques – Elasticsearch provides powerful aggregation (group‑by) capabilities. A bucket groups documents (e.g., by color ), while a metric computes statistics (count, sum, avg, min, max) on each bucket.

Example: Group cars by color and compute average price:

GET /cars/_search
{
  "aggs": {
    "group_by_color": {
      "terms": { "field": "color" },
      "aggs": {
        "avg_by_price": { "avg": { "field": "price" } }
      }
    }
  }
}

Nested aggregations enable drill‑down analyses, such as grouping first by color then by brand , or using date_histogram to aggregate sales per month.

Other useful aggregation patterns include:

Top Hits – Retrieve the highest‑scoring document per bucket.

Histogram – Bucket numeric fields into fixed intervals.

Date Histogram – Bucket date fields by calendar intervals (month, quarter, etc.).

Global Bucket – Compute metrics on the entire index, ignoring the query filter.

Filter Aggregations – Apply additional filters inside aggregations for refined metrics.

These aggregation features allow complex analytics similar to SQL GROUP BY queries, supporting ordering, sub‑aggregations, and combined search‑and‑aggregation workflows.