Two Solutions for ElasticSearch Wildcard Index Collection: Flink ETL Outperforms Logstash/Kafka Pipeline

The author encountered an issue where DataX cannot collect ElasticSearch index data of wildcard type, prompting a dispute with a cloud vendor (Part 2 of the series). To resolve this, two solutions were explored and evaluated.

Solution 1: Logstash → Kafka → DataWorks → MaxCompute

Use Logstash to migrate data from ElasticSearch to Kafka, then leverage DataWorks integration tool to read from Kafka and write into MaxCompute.

Pros: Uses existing, readily available tools.

Cons: Data pipeline is lengthy; requires converting JSON data into two-dimensional tables in MaxCompute, increasing development and maintenance overhead.

Solution 2: Flink ETL Direct Write to MaxCompute

Leverage Flink's ETL capability to periodically read ElasticSearch index data and write it directly into MaxCompute.

Pros: Simpler data pipeline; no need to transform JSON into two-dimensional tables.

Cons: Currently unable to specify target partitions when writing.

Decision

Considering both cost and operational complexity, Solution 2 (Flink ETL) offers clear advantages and is recommended as the preferred approach. The author shares this as a reference for others facing similar wildcard index collection challenges.