Deploy MogileFS with Nginx Reverse Proxy for Scalable File Storage

Introduction

With the explosion of data in the big‑data era, traditional storage cannot scale; distributed file systems such as GFS, HDFS, GlusterFS, etc., address this limitation. This article demonstrates how to build a MogileFS cluster accessed through an Nginx reverse proxy.

Distributed File System

A distributed file system presents a standard file‑system API to clients while storing file contents and directories across multiple machines via the network.

MogileFS Overview

MogileFS is an open‑source distributed file system used by many services (e.g., Yupoo, Digg, Tudou, Douban). Its components include:

Tracker (mogilefsd) : stores global metadata in a database.

Storage nodes (mogstored) : listen on port 7500 and store file chunks.

Utilities (mogadm, mogupload, moglistkeys) .

Client APIs (Perl, PHP).

Implementation Process

Ideal Architecture

Experimental Topology

System Environment

# system environment: CentOS6.6

MariaDB Configuration

Grant privileges for root and MogileFS users:

grant all on *.* to 'root'@'172.16.%.%' identified by 'scholar';

grant all on mogilefs.* to 'moguser'@'172.16.%.%' identified by 'mogpass';

flush privileges;

MogileFS Configuration

Install required packages:

# cd mogilefs/

# yum install MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm

Tracker configuration ( /etc/mogilefs/mogilefsd.conf) example:

daemonize = 1

pidfile = /var/run/mogilefsd/mogilefsd.pid

db_dsn = DBI:mysql:mogilefs:host=172.16.10.211

db_user = moguser

db_pass = mogpass

listen = 0.0.0.0:7001

Storage node configuration ( /etc/mogilefs/mogstored.conf) example:

maxconns = 10000

httplisten = 0.0.0.0:7500

mgmtlisten = 0.0.0.0:7501

docroot = /mogdata

Synchronize configuration files to the second node: # scp /etc/mogilefs/* node2:/etc/mogilefs/ Create device mount points:

# mkdir /mogdata/dev1 -pv

# chown -R mogilefs.mogilefs /mogdata/

Initialize the database:

# mogdbsetup --dbhost=172.16.10.211 --dbrootuser=root --dbrootpass=scholar --dbuser=moguser --dbpass=mogpass --dbname=mogilefs --yes

Start services:

# service mogilefsd start

# service mogstored start

Add Trackers, Devices, Domains

Example commands:

# mogadm --trackers=172.16.10.123:7001 host add node1 --ip=172.16.10.123 --status=alive

# mogadm --trackers=172.16.10.123:7001 device add node1 1

# mogadm --trackers=172.16.10.123:7001 domain add files

Verify domains and upload test files using mogupload and moglistkeys.

Nginx Integration

Compile Nginx with MogileFS Module

Install development tools and dependencies, extract sources, then configure:

# yum groupinstall "Development Tools" "Server Platform Development" -y

# yum install openssl-devel pcre-devel -y

# tar xf nginx_mogilefs_module-1.0.4.tar.gz

# tar xf nginx-1.6.3.tar.gz

# cd nginx-1.6.3

# ./configure \

--prefix=/usr/local/nginx \

--user=nginx \

--group=nginx \

--with-http_ssl_module \

--add-module=../nginx_mogilefs_module-1.0.4

# make && make install

Create Init Script

Place the script at /etc/rc.d/init.d/nginx, make it executable, and add to startup:

# chmod +x /etc/rc.d/init.d/nginx

# chkconfig --add nginx

# chkconfig nginx on

Nginx Configuration

Define upstream trackers and location blocks for each MogileFS domain:

upstream trackers { server 172.16.10.123:7001; server 172.16.10.124:7001; }

server {

listen 80;

location ~*/images/ {

mogilefs_tracker trackers;

mogilefs_domain images;

mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; }

}

location ~*/html/ {

mogilefs_tracker trackers;

mogilefs_domain html;

mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; }

}

location ~*/files/ {

allow 172.16.0.0/16;

deny all;

mogilefs_tracker trackers;

mogilefs_domain files;

mogilefs_methods PUT DELETE;

mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; }

}

}

Test and Start Nginx

# nginx -t

# service nginx start

Verification

Access URLs to retrieve stored files; the system continues to serve data even when one storage node is stopped, demonstrating fault tolerance.

Conclusion

The tutorial shows a functional MogileFS cluster behind an Nginx reverse proxy. Further work can add high‑availability for Nginx and MariaDB.