Elasticsearch Index Design and Sharding Principles

1. Elasticsearch Index Design

Discusses whether to use a single index or time‑based indices, highlighting limitations of a single index such as inability to update mappings, limited scalability, and suitability only for small, static datasets.

1.1 Single vs Time‑Based Index

Time‑based indices require decisions on interval (data volume, change frequency) and implementation via index templates.

1.2 Index Definition Considerations

Provides a concrete index template example with settings, aliases, and strict mappings, and stresses not to define multiple types in one index (unsupported from ES 6.x onward).

Additional best‑practice notes include disabling _source when raw documents are unnecessary, turning off _all for known query fields, using dynamic: "strict" to prevent dirty data, choosing keyword type for exact matches, and employing index aliases for zero‑downtime switches.

2. Elasticsearch Shard Allocation Principles

Answers common community questions about the number of indices, shards, shard size, and replica count, and presents a six‑step practical workflow: define the index, evaluate data volume, estimate index size and disk usage, calculate shard count (keeping each shard <30 GB and scaling with node count), assess the number of indices and types, and iterate as needed.

3. Data Deduplication Strategies

Compares three approaches: using a unique ID (which can increase storage and cause high cardinality), using aggregation to find duplicate hash values, and using distinct queries; includes example aggregation DSL wrapped in GET *_index/_search { ... } and notes on performance and storage trade‑offs.

4. Summary

The notes capture the core insights from an Elasticsearch sharing session held on 2018‑06‑30, offering actionable steps and configuration snippets that can be directly applied in production environments.