How to Deploy a Scalable GlusterFS Cluster with NFS‑Ganesha on Linux

1. GlusterFS Overview

GlusterFS is a scalable network file system that offers high scalability, high availability, high performance, and horizontal expansion without a metadata server, eliminating single points of failure.

When a client accesses GlusterFS storage, it reads and writes data via a mount point, making the cluster file system transparent; operations are handled by the Virtual File System (VFS), passed to the FUSE kernel module, which communicates with the GlusterFS client and ultimately the GlusterFS server over the network.

2. Common GlusterFS Volume Types

Distributed volume (also called hash volume) stores files across multiple bricks using a hash algorithm.

Use case: large numbers of small files.

Advantages: good read/write performance.

Disadvantages: if a brick fails, its data is lost.

Default volume type is distributed; there is no limit on the number of bricks.

Command to create a distributed volume:

gluster volume create volume_name node1:/data/br1 node2:/data/br1

Replica volume stores multiple copies of each file on different bricks.

Use case: scenarios requiring high reliability and high read/write performance.

Advantages: good read/write performance.

Disadvantages: write performance is lower; more storage space is consumed.

Command to create a replica volume (replica count 2):

gluster volume create volume_name replica 2 node1:/data/br1 node2:/data/br1

Stripe volume splits a file into stripes and stores them across multiple bricks (default stripe size 128 KB).

Use case: large files.

Advantages: suitable for large‑file storage.

Disadvantages: low reliability; a brick failure can cause total data loss.

Command to create a stripe volume (stripe count 2):

gluster volume create volume_name stripe 2 node1:/data/br1 node2:/data/br1

Distributed‑replica volume combines hashing and replication, storing multiple copies of each file across different bricks.

Use case: large numbers of files with high reliability requirements.

Advantages: high reliability and high read performance.

Disadvantages: consumes more storage space and has slower write performance.

Command to create a distributed‑replica volume (replica count 2):

gluster volume create volume_name replica 2 node1:/data/br1 node2:/data/br1 node3:/data/br1 node4:/data/br1

Distributed‑stripe‑replica volume stores striped files with multiple replicas, suitable for very large files that also need high reliability.

Use case: ultra‑large files with strict reliability.

Advantages: supports large‑file storage with high reliability.

Disadvantages: higher storage consumption and slower writes.

Command to create a distributed‑stripe‑replica volume (stripe 2, replica 2):

gluster volume create volume_name stripe 2 replica 2 node1:/data/br1 node2:/data/br1 node3:/data/br1 node4:/data/br1

3. GlusterFS Environment Setup

The log storage cluster uses a distributed‑replica volume across five servers (total raw capacity 90 TB, usable 45 TB). Each server hosts two bricks to satisfy the even‑brick requirement.

# sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/sysconfig/selinux   # Disable SELinux
# iptables -F   # Flush firewall rules

Install GlusterFS and related packages on all nodes:

# yum install userspace-rcu-*
# yum install python2-gluster-3.13.2-2.el7.x86_64.rpm
# yum install tcmu-runner-* libtcmu-*
# yum install gluster*
# yum install nfs-ganesha-*
# systemctl start glusterd.service   # Start Gluster daemon on all servers
# systemctl start rpcbind
# systemctl enable glusterd.service
# systemctl enable rpcbind
# ss -lnt   # Verify port 24007 is listening

Peer probe to add nodes to the cluster (run on the admin node 10.102.23.44):

[root@admin-node ~]# gluster peer probe 10.102.23.44
[root@admin-node ~]# gluster peer probe 10.102.23.45
[root@admin-node ~]# gluster peer probe 10.102.23.46
[root@admin-node ~]# gluster peer probe 10.102.23.47
[root@admin-node ~]# gluster peer probe 10.102.23.4

Check peer status: [root@admin-node ~]# gluster peer status Prepare disks on each node (example using /dev/sdb‑/dev/sdk):

# fdisk /dev/sdb   # Partition disks as needed
# vgcreate vg_data01 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
# vgcreate vg_data02 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1
# vgdisplay
# lvcreate -n lv_data01 -L 9TB vg_data01
# lvcreate -n lv_data02 -L 9TB vg_data02
# lvdisplay
# mkfs.xfs /dev/vg_data01/lv_data01
# mkfs.xfs /dev/vg_data02/lv_data02
# mkdir -p /data_01/node /data_02/node
# vim /etc/fstab   # Add entries for the logical volumes
/dev/vg_data01/lv_data01 /data_01 xfs defaults 0 0
/dev/vg_data02/lv_data02 /data_02 xfs defaults 0 0
# mount /data_01
# mount /data_02

Create a distributed‑replica volume (minimum 4 servers required):

# gluster volume create data-volume replica 2 \
    10.102.23.4:/data_01/node 10.102.23.44:/data_01/node \
    10.102.23.44:/data_02/node 10.102.23.45:/data_02/node \
    10.102.23.45:/data_01/node 10.102.23.4:/data_02/node \
    10.102.23.46:/data_01/node 10.102.23.47:/data_01/node \
    10.102.23.46:/data_02/node 10.102.23.47:/data_02/node force

Start the volume and verify:

# gluster volume start data-volume
# gluster volume info
# gluster volume status

4. NFS‑Ganesha Environment Setup

GlusterFS supports NFS mounts, but to serve multiple network segments a proxy is needed; NFS‑Ganesha provides a clean integration by exposing GlusterFS volumes via NFSv4.

Install and configure NFS‑Ganesha on the management node (10.102.23.44). Example excerpt from /etc/ganesha/ganesha.conf:

EXPORT {
    Export_Id = 12345;
    Path = /data01;
    Pseudo = /data01;   # NFS export root
    Protocols = 4;      # Use NFSv4
    Access_Type = RW;   # Read‑write access
    Squash = No_root_squash;
    Sectype = sys;       # Security type
    FSAL {
        Name = GLUSTER;
        hostname = "10.102.23.44";
        volume = "data-volume";
    }
}

Restart and enable the service:

# systemctl restart nfs-ganesha
# systemctl enable nfs-ganesha
# showmount -e 10.102.23.44   # Verify export list

5. Client Mounting

Mount the volume using the native GlusterFS client:

# mkdir /logs
# mount -t glusterfs 10.102.23.44:data-volume /logs/

Mount the same volume via NFSv4:

# yum -y install nfs-utils rpcbind
# systemctl start rpcbind
# systemctl enable rpcbind
# mkdir /home/dwweiyinwen/logs/
# mount -t nfs -o vers=4,proto=tcp,port=2049 10.102.23.44:/data01 /home/dwweiyinwen/logs/