How to Sync MySQL Data to Elasticsearch with Logstash: Step‑by‑Step Guide

1. Preparation

1.1 Install JDK

Refer to online articles such as https://www.dalaoyang.cn/article/16 for installation instructions.

1.2 Install Logstash

Both 5.x and 6.x versions are supported; see https://www.dalaoyang.cn/article/80 for details. Version 7.x has not been tested.

1.3 Install Ruby

Follow the detailed guide at https://blog.csdn.net/qq_26440803/article/details/82717244.

1.4 Install gem

yum install gem

1.5 Modify Logstash Gemfile and Gemfile.jruby-1.9.lock

Update the Gemfile source to the Ruby China mirror:

source "https://gems.ruby-china.com/"

Update the lock file to use the same mirror:

remote: https://gems.ruby-china.com/

1.6 Install bundler

gem install bundler

1.7 Install logstash-input-jdbc plugin

bin/logstash-plugin install logstash-input-jdbc

1.8 Prepare mysql-connector-java-x.x.x.jar

Download the appropriate MySQL connector JAR (e.g., mysql-connector-java-5.1.6.jar) from the official MySQL site.

2. Configure Logstash

Create a configuration file (e.g., mysql-es.conf) with the following content:

input {
  jdbc {
    # Driver JAR location
    jdbc_driver_library => "/usr/local/logstash/logstash-5.6.16/mysql-connector-java-5.1.6.jar"
    # Driver class
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    # Database URL
    jdbc_connection_string => "jdbc:mysql://ip:3306/dalaoyang"
    # Credentials
    jdbc_user => "root"
    jdbc_password => "password"
    # SQL query
    statement => "SELECT * from link_info"
    # Paging
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    # Tracking column
    tracking_column => "update_date"
    use_column_value => false
    schedule => "* * * * *"
  }
}
output {
  elasticsearch {
    hosts => "ip:port"
    index => "test-mysql"
    document_id => "%{link_id}"
  }
}

3. Start Logstash

Run the following command: bin/logstash -f mysql-es.conf After starting, you should see Logstash processing logs.

4. Verify in Elasticsearch

Use Elasticsearch‑head or any Kibana/REST client to confirm that the MySQL data has been indexed.

5. Summary

The overall process is straightforward and works without major issues; this is one of many ways to synchronize MySQL data to Elasticsearch.