Using Nomad Constraints to Control DTLE Failover Domains

Background: A ticket reported that DTLE clients are deployed in two data centers, and only DTLE clients can communicate with each other. The goal is to ensure that during a failover the tasks are switched to a client within the same data center.

Because a DTLE client can only access the MySQL instance in its local data center, the failover must schedule the task to a DTLE client that resides in the same data center to maintain connectivity.

Nomad’s documentation shows that the constraint attribute can be used to restrict the range of eligible clients. The article demonstrates how to apply this attribute in a DTLE job.

1. DTLE Job Structure A DTLE job contains two groups, each with a single task that holds the DTLE configuration. These tasks are ultimately deployed on DTLE clients.

2. Constraint Overview The constraint can be defined at the job, group, or task level and consists of three parameters: attribute , operator , and value . The attribute specifies which node property to filter, the operator defines the comparison (default "="), and the value is the target to compare against.

The article lists many node attributes (e.g., ${node.unique.id} , ${node.region} , ${node.datacenter} , ${node.class} , ${attr. } , ${meta. } ) and notes that only ${node.datacenter} , ${node.class} , and ${meta. } are suitable for failover filtering because they can be set as tags on the client.

3. Deploying a DTLE Cluster A cluster with three DTLE clients is created: one client’s datacenter is set to "shanghai" and the other two to "beijing". The DTLE configuration file looks like:

# dtle configuration file
name = "dtle-1"  # rename for each node
 datacenter = "shanghai"
 ...

4. Submitting the DTLE Job The job defines source and destination tasks with constraint attributes set to the appropriate datacenter values ("shanghai" for the source task and "beijing" for the destination task).

job "test_constraint" {
  datacenters = ["shanghai", "beijing"]

  group "Src" {
    constraint {
      attribute = "${node.datacenter}"
      operator  = "="
      value     = "shanghai"
    }
    task "src" {
      driver = "dtle"
      config {
        ReplicateDoDb = [{ TableSchema = "test" }]
        ConnectionConfig = {
          Host = "172.100.9.1"
          Port = 3306
          User = "test_src"
          Password = "test_src"
        }
      }
    }
  }

  group "Dest" {
    constraint {
      attribute = "${node.datacenter}"
      operator  = "="
      value     = "beijing"
    }
    task "dest" {
      driver = "dtle"
      config {
        ConnectionConfig = {
          Host = "172.100.9.2"
          Port = 3306
          User = "test_dest"
          Password = "test_dest"
        }
      }
    }
  }
}

5. Verification and Failover After deployment, the tasks are correctly allocated to nodes with the matching datacenter tags. Stopping the "dtle-2" node causes the destination task to migrate to the remaining "beijing" node as expected. However, stopping the only remaining "beijing" node results in a failure because no suitable datacenter is available.

Conclusion The article demonstrates how to use the datacenter attribute in Nomad constraints to limit DTLE failover domains. It encourages further discussion on using other Nomad attributes and operators for more complex constraint configurations.