From Monolith to Microservices: Lessons Learned and Best Practices

This article introduces microservice architecture and its components, focusing on a high‑level view rather than implementation details.

Understanding microservices starts with recognizing what a monolithic application is; the two are opposite approaches. The transition from monolith to microservices is gradual, illustrated with an online supermarket example.

Initial Requirements

Several years ago, Xiao Ming and Xiao Pi launched an online supermarket. Their initial needs were simple: a public website for browsing and purchasing products, and an admin backend for managing users, products, and orders.

Feature list:

Website

User registration and login

Product display

Order placement

Admin backend

User management

Product management

Order management

The first architecture was a single monolithic deployment (image omitted for brevity).

Business Growth and Emerging Problems

Rapid competition forced the team to add promotions, mobile apps, and data analysis. The resulting ad‑hoc extensions caused many issues:

Duplicate business logic across website and mobile apps

Inconsistent data sharing via databases or APIs, leading to tangled call relationships

Blurred service boundaries and mixed responsibilities

Performance bottlenecks in the admin backend after adding analytics and promotion features

Shared database schema preventing refactoring and causing contention

Deployment and testing became cumbersome; a small change required redeploying the whole system

Team friction over ownership of common functionality

These problems highlighted the need for a more modular design.

Time for Change

The team abstracted common capabilities into independent services:

User Service

Product Service

Promotion Service

Order Service

Analytics Service

Each application now consumes data from these services, eliminating redundant code. The next architecture still used a shared database, so some monolithic drawbacks remained.

To resolve database contention, the team split the persistence layer per service and introduced a message‑queue for real‑time communication (image omitted).

After the split, services could adopt heterogeneous technologies: analytics could use a data warehouse, while product and promotion services added caching.

No Silver Bullet

Even with a cleaner architecture, new challenges appeared:

Fault isolation became harder because a failing service could cascade failures across the system

Overall stability decreased as more services increased the probability of individual failures

Deployment and management overhead grew significantly

Coordinated development and testing across services became more complex

The team addressed these by improving monitoring, tracing, logging, and resilience patterns.

Monitoring – Detecting Early Signs of Failure

High‑concurrency distributed systems need comprehensive metrics. Each component exposes a uniform /metrics endpoint; a collector (Prometheus) scrapes these metrics, stores them, and a UI (Grafana) visualizes them and triggers alerts (image omitted).

Tracing – Pinpointing Faults

Requests often traverse multiple services, so tracing records the call chain using headers: traceId, spanId, parentId, and timestamps. The team adopted Zipkin (an open‑source Dapper implementation) and added an HTTP interceptor to inject these headers and send logs asynchronously (image omitted).

Log Analysis

Large‑scale systems generate massive, distributed logs. The team built a searchable log system using the ELK stack (Elasticsearch, Logstash, Kibana) to aggregate, index, and visualize logs (image omitted).

Gateway – Access Control and Service Governance

A gateway sits between callers and services, enforcing authentication and providing a unified API surface. The team chose a coarse‑grained gateway per service group (image omitted).

Service Registration & Discovery – Dynamic Scaling

Redundancy mitigates failures, but manual registration of new instances is error‑prone. A service‑discovery component (e.g., Consul, Eureka) allows services to self‑register and clients to discover healthy instances automatically (image omitted).

Circuit Breaking, Fallback, and Rate Limiting

Circuit Breaking

When a downstream service repeatedly fails, the circuit breaker opens, returning errors immediately until the service recovers (image omitted).

Service Degradation

Non‑critical features can be degraded to keep core functionality alive during downstream outages.

Rate Limiting

To prevent overload after recovery, services limit request rates globally or per caller (image omitted).

Testing

Testing in microservices occurs at three levels:

End‑to‑end tests covering the whole system (expensive but most confidence)

Service tests for each API

Unit tests for individual code units (fast but limited scope)

Because end‑to‑end tests are costly, they focus on core flows; failures are then traced back to unit tests. Service tests often use mock servers to isolate dependencies (image omitted).

Microservice Framework

To avoid repetitive integration code (metrics, tracing, logging, registration, routing, circuit breaking, etc.), the team built a shared microservice framework that encapsulates these concerns. However, framework upgrades affect all services, requiring careful version management.

Another Path – Service Mesh

Instead of embedding code, a sidecar proxy can handle cross‑cutting concerns. The data plane (sidecars) manages traffic, while the control plane distributes configuration. This approach reduces code intrusion but may add latency (images omitted).

Conclusion – The Journey Continues

Microservice architecture is not the final destination; serverless, FaaS, and even a return to monoliths are possible future directions. The team completed a major refactor and now looks forward to a well‑earned coffee break.