Build a Real‑Time COVID‑19 Dashboard with Elastic Stack and Kibana

Overview

The article explains how to use the Elastic Stack (Elasticsearch 7.6.1, Logstash 7.6.1, Kibana 7.6.1) to build a custom, real‑time COVID‑19 monitoring dashboard. It covers environment setup, data acquisition from public APIs and GitHub repositories, index mapping, data enrichment, and visualization techniques.

Environment

Vagrant base image: bento/centos-8 Elasticsearch 7.6.1

Kibana 7.6.1

Logstash 7.6.1

Data Sources

The primary data set is the DXY COVID‑19 data published at https://ncov.dxy.cn/ncovh5/view/pneumonia. The data is provided as CSV files in the GitHub repository https://github.com/BlankerL/DXY-COVID-19-Data and via an API hosted at https://lab.isaaclin.cn/nCoV/. WHO data can be obtained from the CSSEGISandData repository https://github.com/CSSEGISandData/COVID-19.

Downloading the DXY CSV

git clone https://github.com/BlankerL/DXY-COVID-19-Data.git
cd DXY-COVID-19-Data/
git pull

Future updates can be fetched with git pull and synchronized to Elasticsearch via Logstash or Filebeat.

Index Creation and Mapping

Using Kibana’s built‑in data import UI, the CSV file csv/DXYArea.csv is imported into a new index named dxy-area-m5. The default mapping is replaced with the following custom mapping:

{
  "@timestamp": {"type": "date"},
  "continentEnglishName": {"type": "keyword"},
  "continentName": {"type": "keyword"},
  "countryEnglishName": {"type": "keyword"},
  "countryName": {"type": "keyword"},
  "provinceEnglishName": {"type": "keyword"},
  "provinceName": {"type": "keyword"},
  "province_confirmedCount": {"type": "integer"},
  "province_curedCount": {"type": "integer"},
  "province_deadCount": {"type": "integer"},
  "province_suspectedCount": {"type": "integer"},
  "province_zipCode": {"type": "integer"},
  "cityEnglishName": {"type": "keyword"},
  "cityName": {"type": "keyword"},
  "city_confirmedCount": {"type": "integer"},
  "city_curedCount": {"type": "integer"},
  "city_deadCount": {"type": "integer"},
  "city_suspectedCount": {"type": "integer"},
  "city_zipCode": {"type": "integer"},
  "level": {"type": "keyword"},
  "location": {"type": "geo_point"},
  "is_china": {"type": "boolean"},
  "updateTime": {"type": "date", "format": "yyyy-MM-dd HH:mm:ss"}
}

Data Enrichment with Painless Scripts

Three _update_by_query calls are used to populate the new fields:

# Set is_china flag
POST dxy-area-m5/_update_by_query
{
  "script": {
    "lang": "painless",
    "source": "if (ctx._source.countryEnglishName == \"China\") { ctx._source.is_china = true; } else { ctx._source.is_china = false; }"
  }
}

# Define level (country, province, or cn‑hmt)
POST dxy-area-m5/_update_by_query
{
  "script": {
    "lang": "painless",
    "source": "if (ctx._source.provinceName == ctx._source.countryName) { ctx._source.level = \"country\"; } else { if (ctx._source.cityName == null) { ctx._source.level = \"cn-hmt\"; } else { ctx._source.level = \"province\"; } }"
  }
}

# Normalize province and city names
POST dxy-area-m5/_update_by_query
{
  "script": {
    "lang": "painless",
    "source": "
      if (ctx._source.provinceEnglishName == \"Guangxi\") { ctx._source.provinceEnglishName = \"Guangxi Zhuang Autonomous Region\"; }
      if (ctx._source.provinceEnglishName == \"Hong Kong\") { ctx._source.provinceEnglishName = \"HongKong\"; }
      if (ctx._source.provinceEnglishName == \"Macao\") { ctx._source.provinceEnglishName = \"Macau\"; }
      if (ctx._source.provinceEnglishName == \"Neimenggu\") { ctx._source.provinceEnglishName = \"Inner Mongolia\"; }
      if (ctx._source.provinceEnglishName == \"Ningxia\") { ctx._source.provinceEnglishName = \"Ningxia Hui Autonomous Region\"; }
      if (ctx._source.provinceEnglishName == \"Taiwan\") { ctx._source.provinceEnglishName = \"Taiwan Province\"; ctx._source.provinceName = \"台湾省 (中华人民共和国)\"; }
      if (ctx._source.provinceEnglishName == \"Xizang\") { ctx._source.provinceEnglishName = \"Tibet\"; }
      if (ctx._source.cityName == \"境外输入人员\") { ctx._source.cityName = \"境外输入\"; }
    "
  }
}

After running the scripts, verify the updates with Kibana’s Discovery filters (e.g., countryName:中国, NOT countryName:中国, etc.).

Visualization

Once the data is clean, several visualization approaches are demonstrated:

Discovery‑to‑Visualize: Select a field (e.g., provinceName) in Discovery, click Visualize , set Y‑axis to province_confirmedCount max, and filter by is_china:true and level:province to obtain a province‑level ranking.

Lens: Create a new Lens visualization, choose the index pattern dxy-area-m5, drag fields such as continentName, countryName, province_confirmedCount onto axes, and let Lens suggest chart types.

TSVB (Time Series Visual Builder): Use the TSVB panel to plot cumulative confirmed and cured counts, adjusting metrics and panel options to assess recovery speed.

Maps: Add an EMS Boundaries layer for China provinces, join on the provinceEnglishName field, configure tooltips (e.g., name(zh)), and style the fill color by cumulative confirmed cases.

All visualizations can be saved and assembled into a Dashboard for continuous monitoring.

Importing WHO Data with Logstash

The WHO data set from https://github.com/CSSEGISandData/COVID-19 is imported via Logstash:

Install Java 11 and Logstash 7.6.1:

yum install java-11-openjdk-11.0.6.10-0.el8_1.x86_64
rpm -ivh logstash-7.6.1.rpm

Copy Logstash pipeline configuration files to /etc/logstash/conf.d/:

cp logstash-github-covid-19-daily-reports-template.conf logstash-github-covid-19-time-series-template.conf /etc/logstash/conf.d/

Start Logstash and monitor logs:

sudo systemctl start logstash
sudo tail -f /var/log/logstash/logstash-plain.log

Load the index template mapping in Kibana’s Dev Tools, then let Logstash continuously ingest the WHO CSV files into Elasticsearch. After ingestion, the data can be explored in Discovery and visualized alongside the DXY data.

Final Remarks

The workflow demonstrates a complete ELK pipeline: data acquisition, index mapping, field enrichment, verification, visualization, and dashboard assembly. Mastering each component—Elasticsearch, Logstash, Kibana, and Elastic Maps—enables you to build similar real‑time analytics solutions for other domains.