Building a Real‑Time Dashboard Stack: RocketMQ, Flink, Elasticsearch & Kibana – Part 3 (Environment Setup)

Introduction : In real‑time business monitoring such as e‑commerce dashboards or log alerts, traditional batch processing cannot meet latency requirements. The combination of RocketMQ, Flink, Elasticsearch and Kibana provides a full‑stack solution for low‑latency data collection, stream processing, storage and visualization.

Series plan : This is the third article in a series. The first two parts prepared the local environment; this part focuses on the basic environment preparation and the integration logic of the four components.

Scenario and Elasticsearch sink configuration : The author assumes a task that aggregates regional order statistics. The following code builds an asynchronous Elasticsearch sink for Flink, sets connection parameters, defines an element converter that maps Flink rows to Elasticsearch documents, and configures batch size for debugging.

Elasticsearch8AsyncSinkBuilder.<org.apache.flink.types.Row>builder()
    .setHosts(new HttpHost("127.0.0.1", 9200, "https"))
    .setUsername("elastic")
    .setPassword("yLHpzl+zNuQc0UDrRBZ0")
    .allowInsecure() // Development environment skips self‑signed certificate verification; production should use a proper certificate
    .setElementConverter((row, ctx) -> {
        try {
            LOG.info("🔍 [ES Sink] ElementConverter called, stat_date={}, province={}, city={}",
                row.getField(0), row.getField(1), row.getField(2));
            java.util.Map<String, Object> doc = new java.util.HashMap<>();
            doc.put("stat_date", row.getField(0));
            doc.put("province", row.getField(1));
            doc.put("city", row.getField(2));
            doc.put("total_order_count", row.getField(3));
            doc.put("total_gmv", row.getField(4));
            return new IndexOperation.Builder<>()
                .index("order_dashboard")
                .build();
        } catch (Exception e) {
            LOG.error("🔍 [ES Sink] ElementConverter exception", e);
            throw e;
        }
    })
    .setMaxBatchSize(1) // Debug: send each record immediately for observation
    .setMaxTimeInBufferMS(1000L)
    .build();

Verification : The author confirms that MQ data is consumed correctly, the Elasticsearch index receives data, and shows several screenshots of the MQ console, Elasticsearch index view, and simulated business workload.

Kibana dashboard creation : The author describes three steps—creating a data view from the Elasticsearch index, building a dashboard that visualizes the desired business fields, and confirming that the data appears as expected. Screenshots illustrate each step.

After these steps the simulated development is complete, but the author notes that moving to production involves additional challenges such as handling more complex data sources and ensuring comprehensive field coverage.

Production advice and experience :

Component compatibility: legacy and new components coexist (e.g., Kafka and RocketMQ), requiring flexible handling or unified preprocessing, which may increase resource consumption or maintenance cost.

Maintain a unified data contract when the data pipeline becomes long.

Balance maintenance cost against development effort; reuse existing components when possible.

The author also mentions an alternative production solution: ingest business data via MQ, preprocess required fields, log to Alibaba Cloud SLS, and use SLS aggregation to build dashboards.