Pitfall Diary: Practical Lessons on Using Elasticsearch Nested Types

1. Introduction

During a recent e‑commerce search system refactor, the team initially flattened all product attributes into a single Elasticsearch document, which led to incorrect matches in complex queries.

2. Problem Origin

The original MySQL schema stored products, specifications, and prices in separate tables. After copying this structure directly to Elasticsearch, a T‑shirt with red‑L (price 99) and blue‑M (price 89) was stored as an array of objects. Elasticsearch flattened the array, turning color into ["red","blue"] and size into ["L","M"], so a query for "red L" incorrectly matched the blue‑M variant.

The team first tried concatenating all attribute values into a single string, which fixed matching but made aggregation and maintenance costly.

A production incident (customers searching for "red dress" receiving many blue items) highlighted the severity of the issue.

3. Deep Analysis

The core challenge was preserving relational integrity in a search engine that lacks joins. Elasticsearch’s nested type treats each object in an array as an independent document while keeping a parent link, preventing cross‑field contamination.

Official documentation notes that nested queries can be 3‑5× slower than flat queries.

4. Solution Design

The redesign was split into two phases: (1) solve product‑specification queries with nested types, (2) later handle comments, images, etc.

Mapping example (simplified):

PUT /products
{
  "mappings": {
    "properties": {
      "product_id": {"type": "keyword"},
      "product_name": {"type": "text", "analyzer": "ik_max_word", "search_analyzer": "ik_smart"},
      "category": {"type": "keyword"},
      "brand": {"type": "keyword"},
      "variants": {
        "type": "nested",
        "properties": {
          "sku_id": {"type": "keyword"},
          "color": {"type": "keyword"},
          "size": {"type": "keyword"},
          "price": {"type": "scaled_float", "scaling_factor": 100},
          "stock": {"type": "integer"},
          "sales": {"type": "integer"},
          "status": {"type": "keyword"}
        }
      },
      "create_time": {"type": "date"},
      "update_time": {"type": "date"}
    }
  }
}

Key design notes:

Use scaled_float for price to avoid floating‑point precision issues.

Store enumerations as keyword for exact matching.

5. Query Practice

Exact color‑size query using nested and inner_hits:

GET /products/_search
{
  "query": {
    "bool": {
      "must": [
        {"match": {"product_name": "T恤"}},
        {"nested": {
          "path": "variants",
          "query": {
            "bool": {
              "must": [
                {"term": {"variants.color": "红色"}},
                {"term": {"variants.size": "L"}},
                {"term": {"variants.status": "active"}}
              ]
            }
          },
          "inner_hits": {"size": 10, "sort": [{"variants.price": {"order": "asc"}}]}
        }}
      ]
    }
  },
  "_source": ["product_id","product_name","brand","category"]
}

The query returns only the matching nested documents, avoiding the earlier cross‑match problem.

Multi‑condition example (any of several colors and sizes):

GET /products/_search
{
  "query": {
    "bool": {
      "must": [{
        "nested": {
          "path": "variants",
          "query": {
            "bool": {
              "must": [
                {"terms": {"variants.color": ["红色","蓝色"]}},
                {"terms": {"variants.size": ["M","L"]}},
                {"term": {"variants.status": "active"}}
              ]
            }
          },
          "inner_hits": {"size": 50}
        }
      }]
    }
  }
}

Aggregation on nested fields (color count and average price):

GET /products/_search
{
  "size": 0,
  "aggs": {
    "variants_agg": {
      "nested": {"path": "variants"},
      "aggs": {
        "color_count": {
          "terms": {"field": "variants.color", "size": 20},
          "aggs": {"avg_price": {"avg": {"field": "variants.price"}}}
        }
      }
    }
  }
}

6. Performance Optimization

Initial nested queries increased latency from ~50 ms to >200 ms. Optimizations applied:

Move deterministic filters (e.g., category, brand) to the outer filter clause to reduce relevance scoring.

Replace must with filter where scoring is unnecessary.

Cache frequent filter clauses to improve query cache hit rate.

Introduce a redundant color_size field (e.g., "红色-L") for high‑frequency exact matches, allowing simple term queries.

Adjust JVM heap and GC settings, yielding ~20 % latency reduction.

7. Pitfalls Encountered

Update performance: Nested document updates require re‑indexing the whole parent; batch updates mitigated the issue.

Aggregation semantics: Nested aggregations count nested docs, not parent docs, which caused initial mis‑interpretation of color statistics.

Query complexity: Deeply nested queries (four levels) overloaded the cluster; a complexity review process was introduced.

Data consistency: Partial failures during bulk price updates left the index inconsistent, requiring manual rollback.

8. Project Summary

After months of iteration, nested types dramatically improved search accuracy and enabled reliable aggregations such as color distribution and price ranges. However, they are not a silver bullet; large nested arrays or frequent updates may still favor parent‑child or flat designs.

The experience reinforced the need to model data from a search‑centric perspective rather than simply porting relational schemas.