Mastering Mesos, Zookeeper, and Marathon: A Step‑by‑Step Guide to Building a Docker Cluster

1. Overview of Mesos

Mesos is an open‑source distributed resource manager under Apache, often called the kernel of distributed systems. Developed at UC Berkeley AMPLab and later adopted by Twitter.

Key components:

Mesos‑Master – manages frameworks and slaves, allocates resources.

Mesos‑Slave – runs tasks on a node, reports resources.

Framework – e.g., Hadoop, Spark, Kafka; registers via MesosSchedulerDriver.

Executor – runs framework tasks, typically implemented with Docker containers for fast start‑up and isolation.

Resource allocation flow: slaves report CPU and memory, master’s Allocator module distributes resources to frameworks based on policies. An example workflow is illustrated in the resource‑allocation diagram.

2. Zookeeper Basics

Zookeeper is an open‑source coordination service providing configuration maintenance, naming, distributed synchronization, and group services. It underlies Hadoop and HBase.

Roles:

Leader – initiates elections, updates system state.

Follower – handles client requests, participates in voting.

Observer – forwards writes to the leader, does not vote, improves read scalability.

Client – request initiator.

The core protocol is Zab, which provides atomic broadcast. It operates in two modes: recovery (leader election) and broadcast (synchronization). Each transaction receives a monotonically increasing zxid to guarantee ordering.

Server states:

LOOKING – searching for a leader.

LEADING – elected leader.

FOLLOWING – synchronizing with the leader.

Election algorithms: basic Paxos and fast Paxos (default).

Synchronization steps after a leader is chosen:

Leader waits for followers to connect.

Followers send their highest zxid.

Leader determines a sync point.

Leader notifies followers they are up‑to‑date.

Followers resume serving client requests.

3. Marathon Overview

Marathon is a Mesos framework for long‑running services such as web applications. It provides service discovery, a REST API, SSL, and integrates with HAProxy for load balancing.

4. Docker Cluster Practice

Preparation steps include creating three directories for Zookeeper data, generating three Zookeeper configuration files with distinct ports, and adjusting data directories.

Installation of Zookeeper requires Java (1.7‑1.9). The pseudo‑cluster is set up by editing the configuration files and starting each instance, then verifying roles via the Zookeeper client.

5. Mesos Cluster Deployment

Install the Mesosphere repository on both Mesos‑Master and Mesos‑Slave nodes.

Deploy Mesos‑Master and Mesos‑Slave using the provided images (omitted for brevity). Access the Mesos web UI at http://<em>master_ip</em>:5050 to view tasks.

Deploy Marathon, which registers with Zookeeper to locate the Mesos master. Use Marathon’s REST API to launch a Docker container, e.g., an Nginx image. The JSON payload (nginx.json) is submitted via

curl -X POST -H "Content-Type: application/json" -d @nginx.json http://<em>marathon_ip</em>:8080/v2/apps

.

After submission, the container runs on a random host port (e.g., 31011). The Mesos UI and Marathon UI both show the task status, as illustrated in the accompanying screenshots.

Note: The example uses Nginx for demonstration; the Marathon+Mesos+Docker stack is not intended for publicly exposed services without additional security considerations.