Mastering RHCS: Key Components and Practical Commands for Cluster Management

RHCS Core Components

CMAN (Cluster Manager) is a distributed cluster manager that runs on each node, monitors node status, detects failures, and notifies the underlying system to adjust membership.

DLM (Distributed Lock Manager) provides a peer‑to‑peer lock service used by GFS and CLVM to synchronize metadata and volume updates without a dedicated lock server, improving performance and avoiding single‑node recovery bottlenecks.

CCS (Cluster Configuration System) synchronizes the /etc/cluster/cluster.conf XML configuration file across all nodes, ensuring consistent cluster name, node list, resources, services, and fence devices.

FENCE devices prevent split‑brain scenarios by power‑cycling or isolating failed nodes via hardware management interfaces (e.g., IBM RSAII, HP iLO, IPMI, UPS, SAN/NW switches).

Starting and Stopping the RHCS Cluster

To start the cluster, execute on each node in order:

service cman start
service rgmanager start

Start cman first, then rgmanager. To stop the cluster, reverse the order:

service rgmanager stop
service cman stop

If cman fails to stop, verify that the GFS2 filesystem is unmounted and that all rgmanager instances have stopped.

Managing Application Services with clusvcadm

After the cluster is up, services start automatically; otherwise use clusvcadm to enable, disable, restart, or relocate services.

Start a service :

clusvcadm -e -m <service> <node>

# Example: enable webserver on node web1
clusvcadm -e webserver -m web1

Stop a service :

clusvcadm -s -m <service> <node>

# Example: stop mysqlserver on node Mysql1
clusvcadm -s mysqlserver -m Mysql1

Restart a service :

clusvcadm -R -m <service> <node>

# Example: restart webserver on node web1 (command run on any node)
clusvcadm -R webserver -m web1

Relocate a service :

clusvcadm -r -m <service> <target_node>

# Example: move webserver from web1 to web2
clusvcadm -r webserver -m web2

Monitoring RHCS Cluster Status

Three primary tools are used:

cman_tool : Shows node membership, IP addresses, and join times.

# List all nodes
cman_tool nodes -a

clustat : Provides a real‑time view of node and service states. Use -i <seconds> to refresh.

# Refresh every 3 seconds
clustat -i 3

ccs_tool : Manages the cluster configuration file. Example to propagate an updated cluster.conf:

# Update configuration across the cluster
ccs_tool update /etc/cluster/cluster.conf

The tool checks the config_version field; the version must increase for the update to succeed.

Managing and Maintaining GFS2 Filesystems

Key utilities:

gfs2_fsck : Checks and repairs GFS2, similar to fsck.ext3.

# Run a forced check on /dev/sdb5
gfs2_fsck -y /dev/sdb5

gfs2_tool : Queries and modifies filesystem parameters, displays mount info, freezes/unfreezes the filesystem, and lists journals.

# Show mount information
gfs2_tool df /gfs2
# Freeze the filesystem
gfs2_tool freeze /gfs2
# Unfreeze
gfs2_tool unfreeze /gfs2
# List journals
gfs2_tool journals /gfs2

gfs2_jadd : Adjusts the number and size of journals.

# Set journal size to 64M
gfs2_jadd -J 64M
# Increase mountable nodes to 5
gfs2_jadd -j 5 /gfs2

Additional tools such as gfs2_quota (disk quotas) and gfs2_convert (migration from GFS to GFS2) are available but not covered here.