Mastering GlusterFS: Installation, Volume Types, and Management Commands

Terminology

GlusterFS : an open‑source distributed file system.

Brick : a storage unit exported from a server (e.g., SERVER:EXPORT).

Client : a device that mounts a Gluster volume.

Extended Attributes (xattr) : metadata associated with files or directories.

FUSE : userspace file‑system framework allowing non‑privileged users to create file systems.

Geo‑Replication : replicates data across geographically separated clusters.

GFID : a unique 128‑bit identifier for each file or directory.

Namespace : the POSIX mount point exported by a Gluster volume.

Node : a device that hosts one or more bricks.

RDMA : remote direct memory access, bypassing OS for faster data transfer.

RRDNS : round‑robin DNS for load‑balancing across nodes.

Self‑heal : background process that detects and resolves inconsistencies.

Split‑brain : a state where replicas diverge.

Volfile : Gluster configuration file, typically under /var/lib/glusterd/vols/volname.

Volume : a logical collection of bricks.

Installation

# Install dependencies
yum install xfsprogs wget
yum install fuse fuse-libs
# Partition and format a disk for GFS
fdisk /dev/sdb
mkfs.xfs -i size=512 /dev/sdb1
mount /dev/sdb1 /mnt/sdb1
# Install Gluster repository and packages
wget http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/gluster-epel.repo -O /etc/yum.repo.d/glusterfs.repo
yum install glusterfs glusterfs-server glusterfs-fuse glusterfs-geo-replication
# Start/stop the glusterd service
/etc/init.d/glusterd start
/etc/init.d/glusterd stop
# Enable glusterd at boot
chkconfig glusterd on

Cluster Management

# Add a peer to the trusted pool
gluster peer probe host_or_ip
# View peer status
gluster peer status
# Detach a peer
gluster peer detach host_or_ip

Each node generates a UUID; if a node’s IP or hostname changes, re‑run peer probe. To reuse a previously used hostname, detach the node first, then probe again.

Volume Types

Gluster supports several volume layouts, each illustrated below.

Distributed Volume

A distributed volume spreads files across bricks without redundancy. Use external RAID if redundancy is required.

gluster volume create mamm-volume node1:/media node2:/media node3:/media ...

Replicated Volume

Data is stored on multiple bricks; the replica count must match the number of bricks provided.

gluster volume create mamm-volume repl 2 node1:/media node2:/media

Note: brick count must equal replica count; placing multiple bricks on the same node is possible but reduces fault tolerance.

Striped Volume

Files are split into chunks (default 128 KB) and stored across bricks to improve performance.

gluster volume create mamm-volume stripe 2 node1:/media node2:/media

The stripe count must equal the number of bricks.

Distribute‑Replication Volume

Combines replication and distribution: groups of bricks form replicated sets, which are then distributed.

gluster volume create dr-volume repl 2 node1:/exp1 node2:/exp2 node3:/exp3 node4:/exp4

Brick order matters; the total brick count must be a multiple of K (replica factor) times N (distribution factor).

Distribute‑Striped Volume

Similar to distributed‑replication but each group of bricks forms a striped set.

gluster volume create ds-volume stripe 2 node1:/exp1 node1:/exp2 node2:/exp3 node2:/exp4

Brick count must be a multiple of the stripe factor.

Striped‑Replicated Volume

Data is striped, then each stripe is replicated.

gluster volume create test-volume stripe 2 replica 2 server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4

Brick count must equal stripe count × replica count.

Volume Management Commands

# Start / stop / delete a volume
gluster volume start mamm-volume
gluster volume stop mamm-volume
gluster volume delete mamm-volume

# Expand or shrink a volume
gluster volume add-brick mamm-volume [stripe|replica <count>] brick1 ...
gluster volume remove-brick mamm-volume [replica <count>] brick1 ...

# Replace a brick (online migration)
gluster volume replace-brick mamm-volume old-brick new-brick start|pause|abort|status|commit

# Rebalance after expansion/shrink
gluster volume rebalance mamm-volume start|stop|status

# Trigger self‑heal
gluster volume heal mamm-volume               # heal problematic files only
gluster volume heal mamm-volume full          # heal all files
gluster volume heal mamm-volume info

# Set volume options
gluster volume set mamm-volume key value
# Example: view all configurable options at
# https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_managing_volumes.md

Client Mount Options

Native (glusterfs‑fuse)

# Install fuse client
yum install glusterfs-fuse
# Mount the volume
mount -t glusterfs host_or_ip:/mamm-vol /mnt/glusterfs
# Add to /etc/fstab for auto‑mount
echo "localhost:/mamm-vol /mnt/glusterfs glusterfs defaults,_netdev 0 0" >> /etc/fstab

NFS

# Disable the distro’s NFS server if running
service nfs stop
# Start RPC services required by Gluster NFS
service rpcbind start
rpc.statd
# Mount from client
mount -t nfs -o vers=3 host_or_ip:/path /mnt/point

Windows 7 clients need the “Services for NFS” feature enabled.

CIFS (SMB)

# On server, mount GlusterFS to /mnt/gfs
mount -t glusterfs host:/vol /mnt/gfs
# Export via Samba and start smb service
# On Windows client
mount -t cifs //host/path /mnt/point

Performance Monitoring

Profiling

gluster volume profile mamm-vol start
gluster volume profile info
gluster volume profile mamm-vol stop

Real‑time Top

# Show operation counts per brick
gluster volume top mamm-vol {open|read|write|opendir|readdir} brick node1:/exp1 list-cnt 1
# Show read/write performance per brick
gluster volume top mamm-vol read-perf|write-perf bs 256 count 10 brick node1:/exp1 list-cnt 1

Statistics Export

# Dump internal counters
gluster volume statedump mamm-vol
# Set dump path
gluster volume set server.statedump-path /var/log/
# Export dump to file
gluster volume info dumpfile

Troubleshooting

Gluster uses several ports; firewall rules (iptables) may block peer probing or data sync. Use gluster volume log rotate to rotate logs. After adding bricks, the remote glusterd may temporarily reject connections—retry after a short pause.

References

Official documentation: https://github.com/gluster/glusterfs/tree/master/doc/admin-guide/en-US/markdown

GlusterFS research article: http://blog.csdn.net/liuaigui/article/details/6284551

Comprehensive tutorial: http://www.slashroot.in/gfs-gluster-file-system-complete-tutorial-guide-for-an-administrator

Video introductions: http://edu.51cto.com/lesson/id-35359.html and http://edu.51cto.com/course/course_id-1930-page-2.html

Infiniband guide: http://www.ibm.com/developerworks/cn/linux/l-cn-infiniband/