Step‑by‑Step Guide to Syncing Canal Data to Elasticsearch

This guide walks you through the entire process of using Alibaba Canal to capture MySQL binlog events and synchronizing them to an Elasticsearch cluster.

Environment preparation : Ensure a MySQL instance with binlog enabled, a running Canal server, and Elasticsearch/Kibana services are available.

Install the Canal client‑adapter using Docker:

docker pull slpcat/canal-adapter:v1.1.5

Copy the container's /opt/canal-adapter/conf directory to a local path for easy editing:

docker cp b2f0d149c5c5:/opt/canal-adapter /Users/niu/docker/canal-adapter/conf

Configure the adapter (client‑adapter application.yml ) to set the server port, Canal connection mode, data source, and output adapters. Example snippet:

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
canal.conf:
  mode: tcp
  flatMessage: true
srcDataSources:
  niuDS:
    url: jdbc:mysql://localhost:3306/mycanal?useUnicode=true
    username: canal
    password: canal
canalAdapters:
  - instance: mycanal
    groups:
      - groupId: g1
        outerAdapters:
          - name: logger
          - name: es7
            key: canal-es-key
            hosts: localhost:9200
            properties:
              mode: rest
              cluster.name: elasticsearch

Configure the Elasticsearch adapter ( es7/application.yml ) to map MySQL tables to ES indices:

dataSourceKey: niuDS
outerAdapterKey: canal-es-key
destination: mycanal
groupId: g1
esMapping:
  _index: canal-es
  _id: _id
  upsert: true
  sql: "select id as _id, name, age, ctime, utime from student"
  etlCondition: "where id>2"
  commitBatch: 3000

Create the ES index and mapping :

PUT /canal-es
{
  "settings": {"number_of_shards": "1", "number_of_replicas": "0"},
  "mappings": {
    "properties": {
      "id": {"type": "integer"},
      "name": {"type": "text"},
      "age": {"type": "integer"},
      "ctime": {"type": "date"},
      "utime": {"type": "date"}
    }
  }
}

Start the adapter with Docker, mounting the local configuration directory:

docker run --name canal-adapter -p 8081:8081 \
  -v /Users/niu/docker/canal-adapter/conf:/opt/canal-adapter/conf \
  -d slpcat/canal-adapter:v1.1.5

Troubleshooting covers common issues such as:

Connection refused to Canal server – ensure the Canal service is running and the IP/port match.

CRUD changes not appearing in ES – verify meta.dat cleanup and correct destination settings.

ES node unavailable – check the mode (transport vs. rest) and that ES is reachable at localhost:9200 .

Mapping errors – use a compatible Canal version (≤1.1.5) with Java 8.

Timestamp conversion – remove custom date format in ES mapping to let Canal handle conversion.

Extension loading failures – adjust the plugin directory path.

By following these steps and addressing the listed errors, you can achieve reliable, real‑time synchronization from MySQL via Canal to Elasticsearch.