Ensuring Elasticsearch Stability: Testing, Performance, and Disaster Recovery

Background

Elasticsearch (ES) is an open‑source, distributed, RESTful search engine built on Lucene. Because many core user flows rely on ES, its stability is critical. Common pain points include performance evaluation before release, data accuracy during index creation, real‑time sync latency, rapid recovery after index issues, and efficient debugging of search queries.

Guarantee System

The ES reliability framework is divided into three stages: pre‑release testing, runtime fault detection, and post‑fault handling.

1. Data Validation

Ensuring data accuracy when migrating to ES by comparing business database records with ES documents. Typical issue: optimistic‑lock conflicts causing inconsistent data. Solution: serialize write tasks to avoid concurrent writes to the same index.

2. Performance Testing

Performance tests assess read latency under various cluster configurations, index sizes, and query complexities. Typical issue: cluster performance imbalance due to hardware differences, leading to chain‑reaction problems. Methods to identify hot nodes include swapping data between nodes and adjusting replica counts.

Optimization includes adjusting cluster settings, reducing index size, using script queries instead of large terms, and splitting terms into smaller batches.

3. Data Delay Alarm

Distinguish offline sync (low latency requirement) from real‑time sync (high latency sensitivity). Set up scheduled tasks to fetch recent business increments and verify their presence in ES; trigger alerts when delays are detected.

4. Real‑time Data Correction

When the data sync system fails, a corrective program writes missing data directly to ES. Key points: handle optimistic‑lock conflicts, use upsert to avoid overwriting fields, map core fields correctly, and obtain increments via change listeners.

5. Degradation Measures – Disaster Recovery

Potential issues: node failures, index generation errors. Measures: multi‑cluster backup, index alias switching, retaining recent indices, and using backup clusters for failover.

# Switch index alias
POST /_aliases
{
  "actions": [
    { "remove": { "index": "floorplan_prod_20250413", "alias": "floorplan" } },
    { "add":    { "index": "floorplan_prod_20250414", "alias": "floorplan" } }
  ]
}

6. ES Version Debug Platform

A debugging tool links search requests with target IDs to pinpoint unsatisfied conditions, greatly speeding up issue localization without manual query inspection.

Conclusion and Outlook

The described testing and mitigation techniques have prevented serious production incidents, handling over five major issues without escalation. Future work includes leveraging ES’s KNN vector search for AI‑driven search quality assurance.