Why Our Microservice Overhaul Sparked Explosive Complexity—and What We Learned

Background Introduction

The company provides data services to over 20,000 customers via APIs that collect and clean client data. After a microservice transformation, the system grew to 400 private repos and 70 different services (workers).

Benefits of the Microservice Refactor

Improved visibility and monitoring with tools such as sysdig, htop, and iftop.

Significantly reduced configuration and deployment costs.

Avoided the temptation to add disparate features to existing services.

Created many low‑dependency services that simply read from queues, process data, and send results.

Facilitated small‑team collaboration.

Monitoring each microworker with Datadog made issue isolation easier, allowing memory‑leak problems to be narrowed down to 50‑100 lines of code.

Microservice Architecture Overview

The system receives hundreds of thousands of events per second and forwards them to partner destinations (e.g., Google Analytics, Optimizely, custom webhooks). Initially a simple API handled events and queued them. Over time, the number of destinations grew to over 100, each with its own service.

Failures are categorized as retryable (e.g., HTTP 500, rate limits, timeouts) or non‑retryable (e.g., invalid credentials, missing required fields). Mixing fresh events with multiple retries in a single queue caused "head‑of‑queue" blocking, increasing latency when a destination slowed down.

To mitigate this, the team introduced a router process that duplicated incoming events to separate queues per destination, isolating failures to the affected service only.

Problems Encountered

Shared‑library version proliferation: 50 new destinations required shared libraries, leading to divergent versions across services and heavy testing/deployment overhead.

Load‑pattern variability: Some services handle few events daily, others process thousands per second, causing manual scaling during unexpected spikes.

Scaling‑tuning difficulty: Diverse CPU and memory requirements made autoscaling configuration more art than science, with the number of services growing rapidly.

Management overhead: Over 140 services strained the team, leading to sleep‑deprived engineers handling peak loads.

Return to Monolith

Facing the complexity, the team merged services back into a monolith, adding a "Centrifuge" component to route events. This reduced the number of repos and queues, improving deployment speed and resource utilization, though it reintroduced challenges such as fault isolation, cache efficiency loss, and dependency version impacts.

Summary

Introducing microservices and isolating destinations solved pipeline performance issues, but bulk updates without proper testing tools caused a rapid decline in developer productivity. Architectural choices are trade‑offs; one must evaluate added complexity, operational cost, scaling control, and management overhead.

Architecture Design Pitfalls

Blindly chasing patterns and principles without assessing real needs.

Following trends without solving actual problems.

Trying to address every concern simultaneously, leading to unfocused designs.

Ignoring architectural decay over the software lifecycle.