Deploying GlusterFS Distributed Replicated Volumes with NFS‑Ganesha

1. GlusterFS Overview

GlusterFS is a scalable network file system that offers high extensibility, availability, and performance without a single metadata server, eliminating single points of failure. Clients access data via mount points; the VFS forwards operations to the FUSE kernel module, which communicates with the GlusterFS client and then the GlusterFS servers over the network.

2. Common GlusterFS Volume Types

Distributed volume (hash‑based) stores files across multiple bricks, ideal for large numbers of small files. It provides good read/write performance, has no limit on brick count, but data on a failed brick is lost.

gluster volume create volume_name node1:/data/br1 node2:/data/br1

Replica volume stores multiple copies of each file on different bricks, improving reliability and read performance at the cost of write speed and storage space.

gluster volume create volume_name replica 2 node1:/data/br1 node2:/data/br1

Stripe volume splits files into stripes across bricks (default stripe size 128 KB), suited for large files but vulnerable to brick failures.

gluster volume create volume_name stripe 2 node1:/data/br1 node2:/data/br1

Distributed‑replica, distributed‑stripe, and stripe‑replica volumes combine the above characteristics for specific workloads, such as high‑throughput large‑file storage with redundancy.

3. Environment Preparation

The deployment uses five servers with a total of 90 TB raw disk space; a distributed‑replica volume provides 45 TB usable space. Two bricks are created on each server to satisfy the even‑brick requirement.

# Disable SELinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/sysconfig/selinux
# Flush iptables rules
iptables -F
# Install GlusterFS and dependencies
yum install userspace-rcu-*
yum install python2-gluster-3.13.2-2.el7.x86_64.rpm
yum install tcmu-runner-* libtcmu-*
yum install gluster*
yum install nfs-ganesha-*
# Start services on all nodes
systemctl start glusterd.service
systemctl start rpcbind
systemctl enable glusterd.service
systemctl enable rpcbind
ss -lnt   # verify port 24007 is listening

Peer nodes are probed to form the cluster:

gluster peer probe 10.102.23.44
gluster peer probe 10.102.23.45
gluster peer probe 10.102.23.46
gluster peer probe 10.102.23.47
gluster peer probe 10.102.23.4

Physical disks are partitioned, volume groups created, logical volumes sized to 9 TB each, formatted with XFS, and mounted:

fdisk /dev/sdb
vgcreate vg_data01 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
vgcreate vg_data02 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1
vgdisplay
lvcreate -n lv_data01 -L 9TB vg_data01
lvcreate -n lv_data02 -L 9TB vg_data02
lvdisplay
mkfs.xfs /dev/vg_data01/lv_data01
mkfs.xfs /dev/vg_data02/lv_data02
mkdir -p /data_01/node /data_02/node
vim /etc/fstab   # add entries for the LV devices
mount /data_01
mount /data_02

Finally, a distributed‑replica volume is created (minimum four servers required) and started:

gluster volume create data-volume replica 2 \
  10.102.23.4:/data_01/node 10.102.23.44:/data_01/node \
  10.102.23.44:/data_02/node 10.102.23.45:/data_02/node \
  10.102.23.45:/data_01/node 10.102.23.4:/data_02/node \
  10.102.23.46:/data_01/node 10.102.23.47:/data_01/node \
  10.102.23.46:/data_02/node 10.102.23.47:/data_02/node force
gluster volume start data-volume
gluster volume info
gluster volume status

4. NFS‑Ganesha Setup

Because some network segments cannot reach the GlusterFS servers directly, NFS‑Ganesha is deployed on the management node (10.102.23.44) to proxy NFSv4 traffic. The configuration file defines an export that maps the GlusterFS volume to an NFS path.

vim /etc/ganesha/ganesha.conf
...
EXPORT {
  Export_Id = 10;
  Path = /data01;
  Pseudo = /data01;   # NFS client root
  Protocols = 4;     # NFSv4 only
  Access_Type = RW;
  Squash = No_root_squash;
  Sectype = sys;
  FSAL {
    Name = GLUSTER;
    hostname = "10.102.23.44";
    volume = "data-volume";
  }
}
...
systemctl restart nfs-ganesha
systemctl enable nfs-ganesha
showmount -e 10.102.23.44   # verify export

5. Client Mounting

Clients can mount the storage either directly via GlusterFS or through NFS‑Ganesha.

# GlusterFS mount
mkdir /logs
mount -t glusterfs 10.102.23.44:data-volume /logs

# NFS mount (from a different subnet)
yum -y install nfs-utils rpcbind
systemctl start rpcbind
systemctl enable rpcbind
mkdir /home/dwweiyinwen/logs
mount -t nfs -o vers=4,proto=tcp,port=2049 10.102.23.44:/data01 /home/dwweiyinwen/logs