Scaling from 20K to 1M Users: Service‑Size Strategies and Low‑Cost Refactoring

Background and Motivation

Initial system: Serverless Node.js backend, 2 Android, 2 iOS, 3 frontend engineers, supporting 200 k users. Problems: single point of failure, cost inefficiency at high traffic, lack of observability, high architectural coupling leading to frequent refactoring.

Service Size Taxonomy

Definitions

Service size is defined by business functionality granularity, not lines of code. Hierarchy (from smallest to largest): Picoservice ≤ FAAS ≤ Microservice ≤ Monoservice ≤ Polyservice. A Polyservice contains the whole application in a single repository; a Picoservice treats each function as an independent service, realistic only on Function‑as‑a‑Service platforms.

Size is independent of deployment scale; any size can be horizontally scaled.

Service Dependency Patterns

Serial ABC : A → B → C. Keep unrelated functions separate to avoid coincidental cohesion.

Mutual AB : A ↔ B. With two nodes splitting is optional.

Tree ABC : A splits into B and C (or reverse). Typical call‑graph tree.

Loop ABCD : A → B ↔ D ← C. Common in callback scenarios.

Real‑world call graphs are composed of these patterns.

Minimal Business Unit (MBU) Pattern

MBU treats each business domain (e.g., article CRUD) as a self‑contained service that can be duplicated for new features. This deliberately violates DRY to reduce vertical complexity and lower bug‑introduction risk.

Benefits : isolates failures, cheap to implement, replaces vertical complexity with manageable horizontal complexity.

When to use : experimental features, frequent rewrites, limited engineering resources, or when rapid isolation of changes is required.

Implementation guidelines: keep each MBU small enough to be replaceable; use a bridge SDK to access legacy data during migration.

Practical Migration Strategy

API Proxy

Built an api-proxy with OpenResty (NGINX + Lua). It includes a Lua LRU cache ( lua-resty-lrucache) and achieved ~80 k QPS on an 8‑core, 16 GB machine with ~20 ms latency on cache hits.

Proxy forwards traffic between the new MBU‑based platform and the old Serverless SDK, enabling staged migration.

User Data Migration

Unregistered users : register directly via the new user‑center API.

Registered users : on first successful login through the proxy, re‑encrypt the password using the new scheme and store it locally.

Inactive users : remain on the proxy for a two‑week transition, then force a password reset.

Deployment Model

Each server runs the full set of services (monolithic per‑machine). Scaling is achieved by adding identical machines behind a reverse proxy. Empirical limit: ~8 repos per CPU core (≈128 repos on a 16‑core box).

To reduce latency from many internal calls, the proxy aggregates required data into a global cache, trading memory for reduced round‑trip time.

Results and Lessons Learned

The refactor lasted six months (Dec 2016 – May 2017). The team shrank to a few engineers while migrating all services using the MBU pattern, selective caching, and staged traffic switching. Key takeaways:

Architectural elegance must align with team capability and business constraints.

Incremental migration with a bridge layer (SDK + API proxy) mitigates risk.

Horizontal scaling of many small services is feasible when each machine hosts a bounded number of repositories.

Deliberate duplication (MBU) can be preferable to complex shared code in environments with limited engineering resources.