How to Build Stable SaaS Systems: Key Practices for Reliability

1. Background of Stability

Stability is critical in ERP SaaS because failures can disrupt business processes for dozens or hundreds of users, affecting modules such as orders, inventory, and finance, which can cause financial loss.

2. Scope of Discussion

The focus is on stability problems caused by service resource fluctuations—memory overflow, CPU spikes, high concurrency—that are visible to users. Bugs in program code are excluded.

3. Middleware Stability (Database, Cache, MQ)

Companies lacking strong technical depth are advised to use cloud services that provide high availability (four‑9 or five‑9). Proper capacity assessment and monitoring of resource‑varying scenarios are essential.

For databases, employing a DBA to manage operations is recommended. Different business scenarios may use various databases such as Alibaba Cloud RDS, PolarDB, PostgreSQL, allowing specialists to handle each workload.

To improve database stability, cache short, CPU‑intensive SQL, optimize or split long, IOPS‑heavy SQL, and use multi‑master‑multi‑slave architectures for caches and queues.

4. Software Iteration Process (Pre‑Release)

SaaS products iterate quickly, often delivering a feature within one or two weeks. Because new features may expose problems, a gray‑release strategy is used: the feature is first exposed to a small user group.

During traffic shifting, the team must ensure that users are not performing critical operations and preferably perform the switch during low‑traffic periods.

5. Software Release Process (During Release)

ERP releases involve dozens of engineering components. Two key principles: no users should be present during the release, and the release should be automated to avoid human errors such as missing code, omitted features, wrong release order, or incorrect branches.

6. Software Runtime Process (Post‑Release)

This stage is the most difficult to control because failures can occur at any node at any time, often before the team can react.

6.1 Establish Comprehensive Monitoring

Monitoring is the foundation. It is divided into system monitoring (node failures, memory shortage, CPU spikes, high concurrency, slow responses) and business monitoring (log analysis via ELK, detecting large DB query results, excessive request parameters, slow services, thread‑pool overflow). Automated alerting is required as monitoring scenarios expand.

6.2 Monitor and Limit Database Query Results

Most service crashes stem from memory overflow caused by excessively large DB result sets. Monitor result sizes at the iBatis or JDBC layer and log them. When necessary, limit results using SQL LIMIT or JDBC maxRows, after evaluating business tolerance.

6.3 Service Load‑Balancing Settings

Default random or round‑robin load balancing (e.g., Nginx, micro‑service frameworks) works for normal cases but can cause delays when a node is unhealthy yet still receives traffic. Use a “least‑active” strategy (available in Dubbo and Nginx) where the client tracks each server’s concurrent requests and avoids routing to overloaded nodes.

6.4 Service Degradation

When external services fail, evaluate business impact and degrade gracefully, for example by catching exceptions or using Dubbo’s mock feature so that downstream processes continue.

6.5 Rate Limiting and Circuit Breaking

Abnormal traffic can affect HTTP requests, inter‑service calls, messaging, or databases. Circuit‑breaker and rate‑limiting tools such as Hystrix and Sentinel are recommended; Sentinel generally offers stronger availability.

6.6 Chaos Engineering

Introduce chaos engineering to deliberately disrupt middleware or inter‑service connections, initially via manual fault injection, to identify system fragilities and reduce the blast radius of real failures.

6.7 SRE Mechanism

Building an SRE team requires senior leadership support and cross‑team collaboration. A mature SRE practice depends on solid monitoring (e.g., Alibaba Cloud ARMS) and encourages engineers to proactively identify and improve stability risks rather than merely documenting incidents. An effective SRE process can quickly raise overall system stability, as illustrated in the diagram.