How to Deploy GlusterFS Distributed Replicated Volumes with NFS‑Ganesha

1. GlusterFS Overview

GlusterFS is a scalable network file system offering high scalability, high availability, high performance, and horizontal expansion without a metadata server, eliminating single points of failure.

Clients access GlusterFS via mount points; reads and writes are handled by the VFS, which forwards requests to the FUSE kernel module and then to the GlusterFS client, ultimately reaching the GlusterFS server over the network.

2. Common GlusterFS Volume Types

Distributed (hash) volume stores files across multiple bricks using a hash algorithm; suitable for many small files, offers good read/write performance, but data on a failed brick is lost.

Replica volume stores multiple copies of each file on different bricks; ideal for high reliability and performance scenarios, but write performance is lower and storage space is increased.

Stripe volume splits files into stripes across bricks (default stripe size 128 KB); suited for large files, provides high concurrency but lacks redundancy.

Distributed‑replica, distributed‑stripe, and stripe‑replica volumes combine the above characteristics for various use cases.

3. GlusterFS Environment Setup

The log storage cluster uses a distributed replica volume across five servers (total raw capacity 90 TB, usable 45 TB). Each server creates two bricks to meet the even‑brick requirement.

# sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/sysconfig/selinux  # disable SELinux
# iptables -F  # flush firewall rules

Install GlusterFS and related packages:

# yum install userspace-rcu-*
# yum install python2-gluster-3.13.2-2.el7.x86_64.rpm
# yum install tcmu-runner-* libtcmu-*
# yum install gluster*
# yum install nfs-ganesha-*
# systemctl start glusterd.service
# systemctl start rpcbind
# systemctl enable glusterd.service rpcbind
# ss -lnt  # verify port 24007 is listening

Create the cluster by probing peers:

[root@admin-node ~]# gluster peer probe 10.102.23.44
[root@admin-node ~]# gluster peer probe 10.102.23.45
[root@admin-node ~]# gluster peer probe 10.102.23.46
[root@admin-node ~]# gluster peer probe 10.102.23.47
[root@admin-node ~]# gluster peer probe 10.102.23.4

Check peer status: [root@admin-node ~]# gluster peer status Prepare disks, create volume groups, logical volumes, format them with XFS, and mount:

# fdisk /dev/sdb
# vgcreate vg_data01 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
# vgcreate vg_data02 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1
# lvcreate -n lv_data01 -L 9TB vg_data01
# lvcreate -n lv_data02 -L 9TB vg_data02
# mkfs.xfs /dev/vg_data01/lv_data01
# mkfs.xfs /dev/vg_data02/lv_data02
# mkdir -p /data_01/node /data_02/node
# echo '/dev/vg_data01/lv_data01 /data_01 xfs defaults 0 0' >> /etc/fstab
# echo '/dev/vg_data02/lv_data02 /data_02 xfs defaults 0 0' >> /etc/fstab
# mount /data_01
# mount /data_02

Create a distributed replica volume (requires at least four servers):

# gluster volume create data-volume replica 2 \
    10.102.23.4:/data_01/node 10.102.23.44:/data_01/node \
    10.102.23.44:/data_02/node 10.102.23.45:/data_02/node \
    10.102.23.45:/data_01/node 10.102.23.4:/data_02/node \
    10.102.23.46:/data_01/node 10.102.23.47:/data_01/node \
    10.102.23.46:/data_02/node 10.102.23.47:/data_02/node force

# gluster volume start data-volume

Verify the volume:

# gluster volume info
# gluster volume status

4. NFS‑Ganesha Setup

GlusterFS supports NFS3 natively, but to serve NFS4 across different network segments, NFS‑Ganesha is used as a proxy. Configure /etc/ganesha/ganesha.conf to export the GlusterFS volume via the VFS FSAL:

EXPORT {
    Export_Id = 10;
    Path = /data01;
    Pseudo = /data01;
    Protocols = 4;
    Access_Type = RW;
    Squash = No_root_squash;
    Sectype = sys;
    FSAL {
        Name = GLUSTER;
        hostname = "10.102.23.44";
        volume = "data-volume";
    }
}

Restart and enable the service:

# systemctl restart nfs-ganesha
# systemctl enable nfs-ganesha
# showmount -e 10.102.23.44

5. Client Mounting

Mount via GlusterFS:

# mkdir /logs
# mount -t glusterfs 10.102.23.44:data-volume /logs/

Mount via NFS4:

# yum -y install nfs-utils rpcbind
# systemctl start rpcbind && systemctl enable rpcbind
# mkdir /home/dwweiyinwen/logs/
# mount -t nfs -o vers=4,proto=tcp,port=2049 10.102.23.44:/data01 /home/dwweiyinwen/logs/