MinIO Cluster Setup: Erasure Coding, S3 Compatibility & Operational Guide

Problem Background

The article begins with three common pain points: single-node MinIO cannot expand storage by adding disks, S3 compatibility gaps cause failures when moving from AWS (multipart upload ETag mismatches, presigned URL host mismatches, unimplemented APIs like ListObjectsV2 pagination, Object Lock retention), and cluster operational challenges (adding nodes, rebalancing, interpreting mc admin info drive health, rolling upgrades, monitoring).

Applicable & Non-Applicable Scenarios

Suitable for: private cloud object storage for images, videos, backups, logs, AI training data; S3 compatibility without public cloud egress fees; 3-16 physical/virtual machines with 4-16 independent data disks each; integration with Prometheus monitoring and existing backup tools (mysqldump, pg_basebackup, Velero).

Not suitable for: single-node <1TB with no reliability needs; POSIX semantics (random writes, file locks); block storage for databases; cross-region strong consistency global namespace; commercial support contracts or LDAP/OIDC SSO (community edition removed external IdP login since 2025-05-24).

Version Baseline & Key Facts

Community edition ( minio/minio, AGPLv3) uses timestamp versions ( RELEASE.YYYY-MM-DDTHH-MM-SSZ), no semantic versioning or LTS. Commercial edition renamed to MinIO AIStor with exclusive features (Iceberg Catalog, SFTP, S3 over RDMA, MemKV). Since RELEASE.2025-05-24T17-08-30Z, embedded Console deprecated, moved to object-browser project; external IdP login removed (moved to AIStor). STS API still works. Community docs now at min.io/docs/minio/community/; AIStor docs at aistor path – commands not interchangeable.

Environment Assumptions (Minimum Production Spec)

4 nodes × 4 disks = 16 disks, erasure code EC:4

Each disk independent mount point, XFS (ext4 not recommended due to O_DIRECT / fdatasync semantics)

Rocky Linux 9 / Ubuntu 24.04, ≥32GB RAM per node

10GbE network, dual-port bond; chrony time sync mandatory (node time diff <15 minutes)

Pre-flight Checks

Kernel ≥5.14 (Rocky 9) or 6.8 (Ubuntu 24.04); glibc ≥2.28 (CentOS 7 fails). File descriptor limits ( ulimit -n and systemd LimitNOFILE). Chrony tracking offset milliseconds, leap status Normal, stratum <5. Disks independently mounted as XFS, no LVM/RAID. MTU 9000 end-to-end consistency (test with ping -M do -s 8972). Ports 9000/9001 free. SELinux permissive or proper contexts. Firewall rules for inter-node 9000, client access via reverse proxy only.

Capacity Planning

Example: need 30TB usable with 20% headroom → 37.5TB usable → raw 75TB at 50% EC:4 efficiency → 21 disks of 4TB (actual ~3.6TB usable) → round to 24 disks (6 nodes × 4). Two pitfalls: vendor 4TB ≠ 3.6TB usable; 20% headroom is minimum because MinIO refuses writes at 100% and deletions may stall. Reserve one node's capacity for heal operations.

Core Knowledge: Erasure Coding, Erasure Sets & Disk Count

6.1 What Erasure Coding Does

MinIO splits objects into data + parity shards across an erasure set . Notation EC:N = N data + N parity = 2N shards, tolerates N simultaneous disk failures. Default EC:4 (4+4=8 shards). Rules: erasure set size power of two (2,4,8,16); each object written to one set (max 16 disks); set membership fixed at startup; EC:4 redundancy 50%; never layer RAID under erasure coding.

6.2 Disk Count Combinations

Table shows total disks vs possible set sizes, sets count, disks per object, fault tolerance, usable capacity (always 50% for EC:4). Counter-intuitive: more "even" disk counts (8,16,24) allow larger sets → higher per-object parallelism → better throughput. But large sets suffer from short-board effect (one slow disk drags whole set). In unstable disk environments, smaller sets give clearer fault boundaries.

6.3 Degraded Mode & "How Many More Can We Lose?"

mc admin info

shows Drives: 15/16 OK and Nodes: 4/4 OK. 15/16 OK means one disk offline; cluster still in read/write quorum. EC:4 per set (8 disks): 1-4 failures read/write; 5 failures read-only; 6+ unrecoverable. Must monitor per-set offline count, not cluster total. Alert rule: any set available disks <6 (for EC:4).

6.4 Storage Classes

MINIO_STORAGE_CLASS_STANDARD=EC:4

, MINIO_STORAGE_CLASS_RRS=EC:2. EC:N means N data + N parity (not RAID6's N+2). RRS reduces parity, not for general use.

6.5 S3 Compatibility Boundaries

Three tiers: (1) Core object ops fully implemented. (2) Behavioral differences: ETag for multipart is MD5(partMD5s)-partCount; checksum support ( x-amz-checksum-*) varies; ListObjectsV2 pagination consistent; strong consistency on XFS/ZFS/btrfs; Object Lock must be enabled at bucket creation; versioning enabled/suspended; event notifications via webhook/AMQP/Kafka/NATS/PostgreSQL/Redis/Elasticsearch (no native Lambda/SQS/SNS); bucket naming rules; region default us-east-1. (3) Not implemented: S3 Select, Object Lambda, Access Points, Transfer Acceleration, cross-account ACL (MinIO uses bucket policies). Critical: if application uses Access Points or Object Lambda, it will fail on MinIO. Generate API dependency list via mc admin trace --verbose --duration 300s.

Implementation Steps

Single-node prep (disks, FS, sysctl, user)

Single-node pseudo-cluster (4 directories) – validates config, mc, systemd, env vars before scaling

4-node cluster (systemd, uniform root credentials)

Load balancer + TLS (Nginx reverse proxy + certs)

S3 compatibility config (users, policies, bucket policies, presigned URLs, CORS, events)

Monitoring & config backup

Pseudo-cluster step isolates variables; avoids debugging four nodes simultaneously.

Directory & User Layout

/data/minio/disk1  ← /dev/nvme0n1
/data/minio/disk2  ← /dev/nvme1n1
/data/minio/disk3  ← /dev/nvme2n1
/data/minio/disk4  ← /dev/nvme3n1

Each disk independent mount, no LVM/RAID. /etc/fstab with UUID, nofail but systemd RequiresMountsFor= to prevent silent degraded start.

Owner minio-user (no login). No pseudo-directories on single disk for production.

System Parameters ( /etc/sysctl.d/99-minio.conf )

net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.ip_local_port_range = 10000 65000
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
fs.file-max = 2000000
fs.inotify.max_user_watches = 524288

Process limits via /etc/security/limits.d/99-minio.conf AND systemd LimitNOFILE=1048576 (systemd ignores limits.conf).

SELinux & Firewall

If enforcing: semanage fcontext -a -t container_file_t "/data/minio(/.*)?", restorecon -Rv /data/minio, setsebool -P httpd_can_network_connect 1, semanage port -a -t http_port_t -p tcp 9000. Or set permissive with audit record. Firewall: allow 9000 between nodes, 9001 only management network.

Install MinIO & mc

Download official binaries, verify SHA256. Do not use distro packages. Record exact version (e.g., RELEASE.2025-10-15T17-29-55Z) – all nodes must match.

Environment File ( /etc/default/minio )

MINIO_VOLUMES="http://minio{1...4}.example.internal:9000/data/minio/disk{1...4}"
MINIO_OPTS="--address :9000 --console-address 127.0.0.1:9001"
MINIO_ROOT_USER="opsadmin"
MINIO_ROOT_PASSWORD="<strong-random>"
MINIO_STORAGE_CLASS_STANDARD="EC:4"
MINIO_BROWSER_REDIRECT_URL="https://console.example.com"
MINIO_SERVER_URL="https://s3.example.com"
MINIO_UPDATE="off"
MINIO_CERTS_DIR=/data/minio/certs

Key: MINIO_VOLUMES uses three-dot expansion {1...4} (not two-dot) so MinIO expands, not shell. Identical on all nodes. Hostnames must resolve consistently. Paths must exist and be empty.

Systemd Unit

Type=notify, RequiresMountsFor all disks, ExecStartPre checks MINIO_VOLUMES, TimeoutStartSec=300, LimitNOFILE=1048576, ProtectSystem=full + ReadWritePaths=/data/minio, OOMScoreAdjust=-500. Certificates in /data/minio/certs (not ~/.minio due to ProtectHome).

Pseudo-cluster Validation

Start

minio server /tmp/minio-test/disk{1..4} --address :9100 --console-address 127.0.0.1:9101

. Verify 4 Online, 0 Offline. Test mc alias, mb, cp, ls, cat, stat. Check mc admin info shows Set: 1, Drives per set: 4. ETag 32-char hex for single PUT. Timestamps match local TZ.

Four-Node Cluster Build

Consistency Check Script

Run on all nodes, compare hostname, minio version, volume string, root user, password hash (sha256), storage class, disks, MTU, NTP offset, kernel max files, SELinux. Any mismatch → stop.

Start & Verify

Restart all four within 1 minute. Expect 16 Online, 0 Offline. If offline: check logs, nc connectivity, chrony. Common errors: backend format mismatch (non-empty dir), credential mismatch, version mismatch, disk unmounted, proxy/firewall.

Bucket & Policy Setup

Enable versioning on app buckets. Object Lock ( --with-lock) at creation only – immutable retention (Compliance mode even root cannot delete). Lifecycle rules for noncurrent version expiration (30 days) and expired delete markers.

Erasure Set Confirmation

mc admin info myminio | grep -iE "pool|set|drive"

. Record actual set size (e.g., two sets of 8). Rules: set size S ∈ {2,4,8,16}, total_disks % S == 0, S % nodes == 0, S/nodes ≥ 1. Snapshot topology weekly.

TLS & Load Balancing

Why Reverse Proxy

Unified endpoint + cert, TLS offload, access control (IP allowlist, rate limit, body size). Must pass X-Forwarded-* headers, support chunked transfer, not rewrite Host header (SigV4 signs Host).

MinIO TLS Config

Certs in /data/minio/certs/public.crt (full chain) and private.key. Set MINIO_CERTS_DIR. MinIO auto-enables HTTPS when certs present.

Nginx Config Highlights

upstream minio_api { keepalive 64; least_conn; server 10.0.10.11:9000 max_fails=3 fail_timeout=10s; ... }
server {
  listen 443 ssl http2;
  server_name s3.example.com;
  ssl_certificate ...;
  ssl_protocols TLSv1.2 TLSv1.3;
  client_max_body_size 0;
  proxy_request_buffering off;
  proxy_buffering off;
  proxy_http_version 1.1;
  proxy_set_header Host $http_host;  # critical
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header Connection "";
  location / { proxy_pass http://minio_api; }
}

Key pitfalls: $http_host not $host (preserves port for SigV4); proxy_buffering off avoids double disk write; client_max_body_size 0 for large objects; HTTP/2 only client→nginx.

Presigned URL Host Issue

Presigned URL signs Host at generation. If generated via internal IP but accessed via public domain → SignatureDoesNotMatch. Solutions: (1) application uses public domain for SDK endpoint; (2) set MINIO_SERVER_URL=https://s3.example.com; (3) avoid proxy rewriting URL.

S3 Compatibility Configuration

User & Access Key Layers

Root (opsadmin) for cluster admin only. Service accounts per app with scoped policies. Human users for audit. Policy JSON: s3:ListBucket resource = bucket ARN; s3:GetObject resource = object ARN ( bucket/*). Must include s3:AbortMultipartUpload and s3:ListMultipartUploadParts for multipart. Never use Resource: "*".

Service Accounts

mc admin accesskey create myminio app-uploader --name ci-uploader --expiry 90d --policy /tmp/policy-ci-narrow.json

. Keys visible once – store in secret manager.

Bucket Policy vs User Policy

Effective permission = union of user policies and bucket policies. Bucket policy can grant anonymous access ( Principal: {"AWS": ["*"]}) – dangerous. Anonymous buckets must have public in name; never on backup/audit buckets.

CORS

Required for browser direct upload. ExposeHeaders must include ETag for multipart completion. AllowedOrigins explicit, not *.

Event Notifications

Targets: Kafka, AMQP, Webhook, NATS, PostgreSQL, Redis, Elasticsearch. Config via mc admin config set notify_kafka:... then mc admin service restart (cluster-wide restart). Bind with mc event add. At-least-once semantics; consumer must be idempotent. Queue limit prevents OOM.

Cross-check with aws-cli

Run list-buckets, put/get small, multipart 20MB, head-object, list-objects-v2 pagination, presign, batch delete. Compare errors: SignatureDoesNotMatch → region/Host; AccessDenied → missing policy actions.

Common Commands & Decision Criteria

Cluster Status

mc admin info

– parse Uptime (node restart?), Version (mismatch?), Network/Drives OK counts, drives healing/offline. Alert on any offline drive >0 for 5min.

Capacity

mc admin info

shows raw capacity. mc du heavy – use Prometheus minio_bucket_usage_total_bytes instead.

Object Ops & Troubleshooting

mc stat

for metadata; mc ls --incomplete for stale multipart; mc rm --incomplete --recursive --force --older-than 7d or lifecycle AbortIncompleteMultipartUpload.

Config Management

mc admin config get/set

. Most api/scanner changes hot-reload; region/volumes require mc admin service restart (cluster restart).

Heal

Trigger after disk replacement, node rejoining, persistent healing drives, read quorum errors. mc admin heal -r --dry-run first. Monitor logs for healed progress.

Troubleshooting: Five Closed-Loop Scenarios

1. Large Upload Fails with XMinioWriteQuorum

Small files ok, large fail at 30-80%. Check offline drives, ongoing heal IO saturation, trace errors. Root causes: disk offline but below read quorum, network jitter, disk IO saturation. Fix: replace disk, restart node, verify 16/16 OK.

2. SignatureDoesNotMatch

mc direct IP ok, domain/proxy fails. Check Host header (nginx $http_host), region consistency, time skew >15min, presigned URL host mismatch. Fix via evidence table.

3. Node Repeatedly Offline

Port listening but cluster shows offline. Check logs for version mismatch, drive unreadable, TLS handshake, backend init, SELinux. Fix: unify versions, remount disks, sync certs, fix SELinux.

4. Delete Objects but Space Not Freed

Causes: versioning retains old versions, delete markers, incomplete multipart. Check mc ls --versions, mc ls --incomplete, mc du --depth 2. Fix: lifecycle NoncurrentVersionExpiration + ExpiredObjectDeleteMarker, abort incomplete multipart.

5. Bucket Delete Fails BucketNotEmpty

Hidden versions/delete markers or multipart. mc ls --versions --recursive, mc rm --recursive --versions --force --dangerous (audit first). Preconditions: no Object Lock, no replication, maintenance window, keep delete manifest.

Production Considerations

Expansion Pitfalls

(1) Adding disks to existing pool fails – erasure sets fixed at startup. (2) Only new node gets new volumes – mismatch. (3) New pool different parity – logs All pools must have the same parity configuration. Correct: add new pool (server pool) with same EC config, update MINIO_VOLUMES on ALL nodes simultaneously via config management, rolling restart. Old data stays; new writes spread by free space. Rebalance optional ( mc admin rebalance start) – heavy IO, run off-peak.

Upgrade Strategy

No LTS. Test target version on staging. Backup IAM & bucket config ( mc admin cluster iam/bucket export). Rolling upgrade: stop node, replace binary, verify checksum, start, confirm rejoined. Keep old binary for rollback (check release notes for irreversible metadata changes).

Monitoring (Prometheus)

Generate scrape config via mc admin prometheus generate. Use dedicated service account for bearer token. Key metrics: minio_cluster_drive_offline_total >0, minio_cluster_capacity_usable_free_bytes <20%, minio_cluster_nodes_offline_total >0, minio_s3_requests_errors_total 5xx>1%, minio_s3_ttfb_seconds p99>1s. Verify metric names per version via curl /minio/v2/metrics/cluster. Set alerts on 2-3× baseline over 3-5 cycles.

Backup Strategy

Four layers: (1) Config (IAM, bucket policies, lifecycle, events) – export daily. (2) Same-city critical buckets – mc mirror --overwrite --preserve daily. (3) Cross-region core buckets – weekly. (4) Ransomware protection – Object Lock Compliance mode on audit/backup buckets. Verify backup by comparing SHA256 of random objects.

Security Hardening Checklist

Root creds 24-char random, not used by apps. Per-app service accounts, least privilege, expiry. Anonymous access default deny, only *-public buckets. Full TLS, disable TLS1.0/1.1. SSE-S3 or SSE-C for encryption at rest. Console only on 127.0.0.1:9001. Audit webhook to external collector with mutual TLS/IP allowlist.

S3 Compatibility Regression Test

Scripted aws-cli test suite: list-buckets, put/get small, content equality (diff not ETag), multipart 20MB, head-object, list-objects-v2 pagination, presign GET, batch delete. Run after every upgrade, proxy change, policy change.

Summary: Four Immutable Rules & Three Operational Habits

EC:N = N data + N parity – 16 disks EC:4 = 50% usable, not 66%.

Expansion only via new pools – cannot add disks to existing pool; old data never auto-migrates; all nodes config change simultaneously via automation.

S3 signing is Host-sensitive – nginx must use proxy_set_header Host $http_host; presigned URLs via MINIO_SERVER_URL or public domain.

Disks independent, XFS, no RAID – RequiresMountsFor prevents silent degradation.

Habits: daily mc admin info check (offline drives, healing); export IAM/bucket config after every change; monthly restore drill (verify backup object hashes, restore test user from IAM export). Final litmus: when someone says "4 disks can fail", ask "in the same erasure set?" – cluster-level numbers hide set-level write unavailability.