Master MinIO: From Client Commands to Scalable Distributed Clusters

1. Introduction

Basic concepts and environment deployment of MinIO can be referenced in the previous article.

Previous article: High‑performance distributed object storage — MinIO (environment deployment): https://www.cnblogs.com/liugp/p/16558869.html

2. MinIO Client (mc) Operations

Official documentation: https://docs.min.io/docs/minio-admin-complete-guide.html

MinIO Client (mc) provides modern replacements for UNIX commands such as ls, cat, cp, mirror, diff, find, etc. It supports file systems and Amazon S3‑compatible cloud storage services (AWS Signature v2 and v4).

alias        set, remove and list aliases in configuration file
ls           list buckets and objects
mb           make a bucket
rb           remove a bucket
cp           copy objects
mirror       synchronize object(s) to a remote site
cat          display object contents
head         display first 'n' lines of an object
pipe         stream STDIN to an object
share        generate URL for temporary access to an object
find         search for objects
sql          run sql queries on objects
stat         show object metadata
mv           move objects
tree         list buckets and objects in a tree format
du           summarize disk usage recursively
retention    set retention for object(s)
legalhold    set legal hold for object(s)
... (additional commands omitted for brevity)
update       update mc to latest release

1) mc Shell Auto‑completion

If you use bash, zsh or fish, shell completion is built into mc. Install it with mc --autocompletion and restart the shell.

# Install
mc --autocompletion
# Restart shell

2) View mc version

mc --version

3) List all buckets from https://play.min.io

mc ls play
# json format
mc --json ls play

4) Create a bucket

The mb command creates a new bucket, similar to mkdir -p . MinIO imposes no limit on the number of buckets per user.

# Add MinIO host
cd /opt/bigdata/minio
./mc config host add minio http://local-168-182-110:19000 admin admin123456

# Create new bucket
mc mb minio/mybucket

# List all buckets
mc ls minio

Web access: http://local-168-182-110:19001

5) Copy files to MinIO

The cp command copies data from one or more sources to a destination.

mc cp wget-log minio/mybucket
# Verify
mc ls wget-log minio/mybucket

6) Daily usage

You can add shell aliases to replace common Unix tools.

alias ls='mc ls'
alias cp='mc cp'
alias cat='mc cat'
alias mkdir='mc mb'
alias pipe='mc pipe'
alias find='mc find'
alias tree='mc tree'

7) MinIO administrator operations

MinIO Client (mc) provides admin sub‑commands for managing a MinIO deployment.

service      restart and stop all MinIO servers
update       update all MinIO servers
info         display MinIO server information
user         manage users
group        manage groups
policy       manage policies defined in the MinIO server
replicate    configure server‑side bucket replication
config       manage MinIO server configuration
... (additional admin commands omitted for brevity)

1) View cluster information

mc admin info minio
# Alias for convenience
alias minfo='mc admin info'
minfo minio

2) Global options

--debug

Example: display detailed debug output for the info command.

mc admin info --debug minio

--json

Enables JSON line‑format output.

mc admin --json info minio

3) service – restart and stop all MinIO servers

mc admin service restart minio

4) policy – manage preset policies

NAME:
  mc admin policy - manage policies

FLAGS:
  --help, -h   show help

COMMANDS:
  add      add new policy
  remove   remove policy
  list     list all policies
  info     show info on a policy
  set      set IAM policy on a user or group

Example: list all preset policies

mc admin policy list minio

Example: add new policy “listbucketsonly”

The policy is stored in /tmp/listbucketsonly.json. When applied, the user can list top‑level buckets only.

Create the JSON file:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListAllMyBuckets"],
      "Resource": ["arn:aws:s3:::*"]
    }
  ]
}

mc admin policy add minio listbucketsonly /tmp/listbucketsonly.json
# Remove the policy
mc admin policy remove minio listbucketsonly

Example: set standard policy “writeonly” on a user

# Add new user
mc admin user add minio newuser newuser123
# Attach policy to user
mc admin policy set minio writeonly user=newuser
# Add group and attach policy
mc admin group add minio somegroup newuser
mc admin policy set minio writeonly group=somegroup

5) user – manage users

Add a new user

mc admin user add minio newuser newuser123

Disable a user

mc admin user disable minio newuser

Enable a user

mc admin user enable minio newuser

Delete a user

mc admin user remove minio newuser

List all users

mc admin user list --json minio

Show user info

mc admin user info minio newuser

6) group – manage groups

Add a user to a group

mc admin group add minio somegroup newuser

Remove a user from a group

mc admin group remove minio somegroup newuser

Delete an empty group

mc admin group remove minio somegroup

Get group info

mc admin group info minio somegroup

List all groups

mc admin group list minio

Enable a group

mc admin group enable minio somegroup

Disable a group

mc admin group disable minio somegroup

7) config – manage server configuration

The config command manages MinIO server configuration.

Get etcd subsystem configuration

mc admin config get minio etcd

Set a specific etcd setting

mc admin config set minio etcd endpoints=http://[hostname|ip]:2379

Export entire server configuration

mc admin config export minio > /tmp/my-serverconfig

Import server configuration

mc admin config import minio < /tmp/my-serverconfig

For more commands, see the official documentation.

3. MinIO Distributed Cluster Expansion

Reference: https://docs.min.io/docs/distributed-minio-quickstart-guide.html

Cluster expansion can be horizontal (adding nodes) or vertical (adding disk capacity). Vertical expansion is not recommended.

1) Peer‑to‑peer expansion

MinIO does not support adding a single node to an existing cluster because data rebalancing and erasure‑coding group changes would be complex. Peer‑to‑peer expansion requires the added nodes and disks to match the original cluster.

Example: a 2‑node, 2‑disk cluster must be expanded by adding 2 nodes with 2 disks each.

After expansion, the original two nodes form one zone and the new two nodes form another zone; new objects are placed according to available space proportion.

Maximum node count is generally limited to 32.

Advantages: simple configuration; a single command can perform the expansion (use consecutive IPs and {} syntax).

Limitations: expansion requires a restart and the node count must stay below 32.

Environment preparation

Hostname: local-168-182-110, IP: 192.168.182.110, data: /opt/bigdata/minio/data/export{1,2}, remark: Original node

Hostname: local-168-182-111, IP: 192.168.182.111, data: /opt/bigdata/minio/data/export{1,2}, remark: Original node

Hostname: local-168-182-112, IP: 192.168.182.112, data: /opt/bigdata/minio/data/export{1,2}, remark: Expansion node

Hostname: local-168-182-113, IP: 192.168.182.113, data: /opt/bigdata/minio/data/export{1,2}, remark: Expansion node

Startup script

#!/bin/bash
mkdir -p /opt/bigdata/minio/logs
mkdir -p /opt/bigdata/minio/data/export{1,2,3,4}
mkdir -p /etc/minio
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=admin123456
/opt/bigdata/minio/minio server --address 0.0.0.0:9000 --console-address 0.0.0.0:9001 --config-dir /etc/minio \
http://192.168.182.11{0...1}/opt/bigdata/minio/data/export{1...2} > /opt/bigdata/minio/logs/minio_server.log

Web access: http://local-168-182-110:19001

Copy configuration to expansion nodes

# On local-168-182-110
cd /opt/bigdata/minio
scp -r /opt/bigdata/minio local-168-182-113:/opt/bigdata/
scp -r /usr/lib/systemd/system/minio.service local-168-182-113:/usr/lib/systemd/system/minio.service

Add disks

# Refresh SCSI bus
for host in $(ls /sys/class/scsi_host); do echo "- - -" > /sys/class/scsi_host/$host/scan; done
lsblk
# Format new disks
mkfs.ext4 /dev/sdb
mkfs.ext4 /dev/sdc
mkfs.ext4 /dev/sdd
mkfs.ext4 /dev/sde
# Create mount points (remove old ones first)
rm -fr /opt/bigdata/minio/data/export{1..4}
mkdir -p /opt/bigdata/minio/data/export{1..4}
# Mount
mount /dev/sdb /opt/bigdata/minio/data/export1
mount /dev/sdc /opt/bigdata/minio/data/export2
mount /dev/sdd /opt/bigdata/minio/data/export3
mount /dev/sde /opt/bigdata/minio/data/export4

Modify startup script for new nodes

#!/bin/bash
mkdir -p /opt/bigdata/minio/logs
mkdir -p /opt/bigdata/minio/data/export{1,2}
mkdir -p /etc/minio
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=admin123456
/opt/bigdata/minio/minio server --address 0.0.0.0:9000 --console-address 0.0.0.0:9001 --config-dir /etc/minio \
http://192.168.182.11{0...1}/opt/bigdata/minio/data/export{1...2} \
http://192.168.182.11{2...3}/opt/bigdata/minio/data/export{1...2} > /opt/bigdata/minio/logs/minio_server.log

Restart services

systemctl restart minio

Web access: http://local-168-182-110:19001

2) Federation expansion

Federation uses etcd to record bucket IP addresses and presents multiple MinIO clusters as a single logical namespace.

etcd configuration files for three nodes (etcd01, etcd02, etcd03) include name, data‑dir, peer URLs, client URLs, cluster token, initial‑cluster, and state.

# etcd01 (local-168-182-110)
name: etcd01
data-dir: /opt/bigdata/etcd/data
initial-advertise-peer-urls: http://192.168.182.110:2380
listen-peer-urls: http://192.168.182.110:2380
listen-client-urls: http://192.168.182.110:2379,http://127.0.0.1:2379
advertise-client-urls: http://192.168.182.110:2379
initial-cluster-token: etcd-cluster
initial-cluster: etcd01=http://192.168.182.110:2380,etcd02=http://192.168.182.111:2380,etcd03=http://192.168.182.112:2380
initial-cluster-state: new

Systemd service file for etcd:

[Unit]
Description=etcd
After=network.target

[Service]
Type=notify
ExecStart=/opt/bigdata/etcd/etcd-v3.4.20-linux-amd64/etcd --config-file=/usr/local/etcd/conf.yml

[Install]
WantedBy=multi-user.target

Start etcd:

systemctl daemon-reload
systemctl start etcd.service

Verify members and health with etcdctl.

Run multiple MinIO clusters

Cluster 1:

cd /opt/bigdata/minio
export MINIO_ETCD_ENDPOINTS="http://192.168.182.110:2380,http://192.168.182.111:2380,http://192.168.182.112:2380"
mkdir -p /etc/minio
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=admin123456
export MINIO_PUBLIC_IPS=192.168.182.110,192.168.182.111
./minio server --address 0.0.0.0:8000 --console-address 0.0.0.0:8001 --config-dir /etc/minio \
http://192.168.182.11{0...1}/opt/bigdata/minio/data/export{3...4}

Web: http://local-168-182-110:8001/

Cluster 2:

cd /opt/bigdata/minio
export MINIO_ETCD_ENDPOINTS="http://192.168.182.110:2380,http://192.168.182.111:2380,http://192.168.182.112:2380"
mkdir -p /etc/minio
export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=admin123456
export MINIO_PUBLIC_IPS=192.168.182.112,192.168.182.113
./minio server --address 0.0.0.0:8000 --console-address 0.0.0.0:8001 --config-dir /etc/minio \
http://192.168.182.11{2...3}/opt/bigdata/minio/data/export{3...4}

Web: http://local-168-182-112:8001/

Configure Nginx as a reverse proxy

# /etc/nginx/conf.d/minio2.conf
upstream minio_api2 {
    ip_hash;
    server 192.168.182.110:8000;
    server 192.168.182.111:8000;
    server 192.168.182.112:8000;
    server 192.168.182.113:8000;
}

upstream minio_console2 {
    ip_hash;
    server 192.168.182.110:8001;
    server 192.168.182.111:8001;
    server 192.168.182.112:8001;
    server 192.168.182.113:8001;
}

server {
    listen 18000;
    server_name 192.168.182.110;
    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
        proxy_ignore_client_abort on;
        proxy_pass http://minio_api2;
    }
}

server {
    listen 18001;
    server_name 192.168.182.110;
    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    location / {
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        chunked_transfer_encoding off;
        proxy_ignore_client_abort on;
        proxy_pass http://minio_console2;
    }
}

Reload Nginx and access via http://local-168-182-110:18001/.

This guide covered basic MinIO operations and practical cluster expansion techniques.