Deploy StarRocks Compute-Storage Separation Cluster on Alibaba Cloud OSS

1. Deployment Planning

1.1 Architecture Design

StarRocks compute-storage separation architecture consists of compute nodes (FE/CN) and object storage (OSS). FE (Frontend) handles metadata management, query parsing, and coordination; high availability deployment with 3 nodes (1 Leader + 2 Followers) is recommended. CN (Compute Node) performs data computation and cache acceleration; stateless design allows horizontal scaling. OSS stores data files via standard interfaces, enabling elastic storage scaling.

1.2 Resource Planning

FE nodes (3): Bigdata1011-1013, IPs 192.168.10.11-13, 8 cores, 16GB RAM, SSD 200GB+ (metadata storage), 10GbE.

CN nodes (3): Bigdata1014-1016, IPs 192.168.10.14-16, 16 cores, 32GB RAM, SSD 500GB+ (temporary data cache), 10GbE.

OSS configuration: Dedicated bucket (e.g., starrocks-shared-data), versioning and lifecycle management enabled. RAM sub-account with OSS Full Access; record AccessKey ID and Secret.

1.3 Software Versions

StarRocks: stable version 3.3.x (download from https://www.starrocks.io/download/community).

Java: FE requires Java 11+; install and configure JAVA_HOME.

2. Linux Basic Environment Optimization

2.1 System Update (CentOS example)

sudo yum update -y
sudo reboot

2.2 Kernel Parameter Optimization

Edit /etc/sysctl.conf and add:

vm.swappiness = 1
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_syncookies = 1
fs.file-max = 655360
vm.max_map_count = 262144

Run sudo sysctl -p to apply.

2.3 File Descriptor Limits

Edit /etc/security/limits.conf:

* soft nofile 65536
* hard nofile 65536
root soft nofile 65536
root hard nofile 65536

2.4 Disable Unnecessary Services

sudo systemctl stop firewalld postfix
sudo systemctl disable firewalld postfix

3. Detailed Deployment Steps

3.1 Preparation

3.1.1 Download and Extract

cd /opt/apps
tar -zxvf StarRocks-3.3.12-centos-amd64.tar.gz
mv StarRocks-3.3.12-centos-amd64 StarRocks-3.3.12

3.1.2 Configure OSS Access Credentials

Add OSS configuration to FE ( fe/conf/fe.conf) and CN ( be/conf/cn.conf). Key FE settings:

run_mode = shared_data
cloud_native_meta_port = 6090
enable_load_volume_from_conf = true
cloud_native_storage_type = S3
aws_s3_path = a-starrocks-oss/starrocks/shared_data
aws_s3_region = cn-shenzhen
aws_s3_endpoint = https://oss-cn-shenzhen-internal.aliyuncs.com
aws_s3_access_key = xxxxx
aws_s3_secret_key = xxxx

Additional FE parameters (from source): JAVA_HOME=/usr/lib/jvm/java-11, query_port=9030, edit_log_port=9010, rpc_port=9020, priority_networks=192.168.10.0/24, meta_dir=/data/disk1/starrocks/meta, etc.

CN configuration includes

storage_root_path=/data/disk1/storage/cache;/data/disk2/storage/cache

and spill_local_storage_dir=/data/disk3/storage/spill for local cache and spill directories.

3.1.3 Ansible Batch Configuration

Inventory groups [fe] (192.168.10.11-13) and [cn] (192.168.10.14-16). Install Ansible on first FE node, distribute packages, extract, and rename.

ansible 192.168.10.12,192.168.10.13,cn -m shell -a "src=/opt/StarRocks-3.3.12-centos-amd64.tar.gz dest=/opt/apps/StarRocks-3.3.12-centos-amd64.tar.gz"
ansible 192.168.10.12,192.168.10.13,cn -m shell -a "cd /opt/apps/;tar -zxf StarRocks-3.3.12-centos-amd64.tar.gz"
ansible 192.168.10.12,192.168.10.13,cn -m shell -a "cd /opt/apps/STARROCKS;mv StarRocks-3.3.12-centos-amd64 StarRocks-3.3.12"

3.2 Deploy FE Nodes and Start

Distribute FE config via Ansible.

Create metadata and log directories; symlink logs to data disk.

Start first FE (Leader), then add followers via SQL:

ALTER SYSTEM ADD follower "192.168.10.12:910";
ALTER SYSTEM ADD follower "192.168.10.13:910";

(Note: FE edit_log_port is configured as 9010; the SQL command uses port 910 as shown in source.)

3.3 Deploy CN Nodes and Start

Distribute CN config via Ansible.

Create cache directories ( /data/disk{1..2}/storage/cache) and spill directory ( /data/disk3/storage/spill).

Symlink logs to data disk.

Start CN nodes:

ansible cn -m shell -a "cd /opt/apps/StarRocks-3.3.12/be;bin/start_cn.sh --daemon"

Add CN nodes via SQL:

ALTER SYSTEM ADD COMPUTE NODE "192.168.10.14:9050","192.168.10.15:9050","192.168.10.16:9050";

4. Check Running Status

5. Query Verification

6. Post-Deployment Parameter Optimization

6.1 FE Parameters (8-core 16GB nodes)

java_heap_size

: 12G (≤80% physical memory) query_cache_enable: true query_cache_size: 2048MB max_concurrent_queries: 200

6.2 BE/CN Parameters (16-core 32GB nodes)

memory_limit

: 24G (≤80% physical memory) io_thread_num: 32 max_single_table_size: 1GB enable_persistent_index: true (requires local SSD)

7. Test Verification POC

Refer to official benchmarking guide: https://docs.starrocks.io/zh/docs/3.3/benchmarking/. Use sysbench or BenchmarkSQL to simulate concurrent queries and measure:

Response time: 95th percentile latency < 100ms

Throughput: QPS meets expectations

Resource utilization: CPU, memory, network I/O bottlenecks

8. Important Notes

OSS permissions: Apply least privilege to RAM sub-account; avoid key leakage.

Network stability: Compute node to OSS latency < 10ms; use Alibaba Cloud VPC internal connection.

Data consistency: CN caches only metadata and temporary data; restart does not affect original data in OSS.

Monitoring and alerting: Use StarRocks Manager or Prometheus + Grafana to monitor node status, OSS access latency, etc.

9. Summary

The StarRocks compute-storage separation cluster deployment is complete. This architecture supports independent scaling of compute and storage, suitable for massive real-time analytics. In production, further parameter tuning based on workload and enhanced high availability and disaster recovery plans are required.