When Microservices Backfire: Lessons from Scaling a Data Service Platform

Background

S Company is a data‑service provider with over 20,000 customers using its software. The company collects and cleans customer data via APIs and offers a suite of products.

Microservices are a mainstream architectural pattern, and S Company refactored its system into microservices, achieving notable benefits.

Refactored scale: 400 private repos; 70 different services (workers).

Benefits achieved:

Visibility – each service can be monitored easily (e.g., sysdig, htop, iftop).

Reduced configuration and deployment costs.

Eliminated the temptation to add disparate functions to existing services.

Created many low‑dependency services that simply read from a queue, process data, and send results, which suits small‑team collaboration.

Problem localization became easier; each microworker can be monitored with Datadog‑style tools.

For example, memory‑leak issues can be narrowed down to 50‑100 lines of code.

In simple terms, microservices are a service‑oriented software architecture where applications are composed of many single‑purpose, low‑overhead network services. Advantages include improved modularity, reduced testing burden, better composability, environment isolation, and team autonomy. Compared with monolithic architecture, where many functions reside in a single unit, microservices enable finer‑grained testing, deployment, and scaling.

Complex, high‑load products often choose microservices for flexibility, strong scalability, and easier monitoring.

However, two years after the refactor, the team did not deliver faster; instead, they faced “explosive” complexity, slower speed, higher failure rates, and team burnout.

System Processing Flow Overview

S Company's customer‑data pipeline receives hundreds of thousands of events per second and forwards them to partner APIs (destinations). Over 100 destination types exist, such as Google Analytics, Optimizely, or custom webhooks.

Initially, a simple architecture used a single API to receive events and forward them to a distributed message queue. Events are JSON objects generated by web or mobile apps, containing user and action information.

When a request fails, it may be retried later. Retryable errors (e.g., HTTP 500, rate‑limit, timeout) can be safely retried; non‑retryable errors (e.g., invalid credentials, missing fields) cannot.

Because a single queue holds both fresh events and multiple retries for all destinations, a “head‑of‑line blocking” problem can occur: if one destination slows down, retries congest the queue, delaying all destinations.

If destination X experiences a temporary timeout, many pending requests pile up and are re‑queued for retry, overwhelming the queue and exceeding auto‑scaling capacity, which leads to latency for new events.

To solve head‑of‑line blocking, the team created separate services and queues for each destination and added a router process that receives inbound events and distributes copies to the selected destinations. Now, only the problematic destination’s queue is blocked, isolating failures.

Problems Encountered

Shared library version proliferation: Adding 50+ new destinations created 50 new repos. Shared libraries were introduced for common transformations, but testing and deploying changes affected all destinations, leading to divergent library versions across repos.

Load pattern issues: Some services handle few events daily, others process thousands per second, requiring manual scaling for low‑traffic destinations during spikes.

Scaling‑tuning challenges: Automatic scaling exists, but each service has different CPU/memory configurations, making tuning more art than science. The number of destinations keeps growing, adding more repos, queues, and services.

Management overhead: With over 140 services, operational overhead became huge, causing sleep‑deprived engineers dealing with load spikes.

Rollback to Monolith

The team eventually abandoned the microservices and merged services back into a single monolith. To avoid checking each destination’s queue, a “Centrifuge” component was added to consolidate all destinations.

All destination code was merged into one repository, allowing a single service deployment. Productivity improved dramatically: a developer could deploy a change in minutes instead of coordinating across 140 services. The unified service mixed CPU‑ and memory‑intensive destinations, simplifying scaling and eliminating the need for paging low‑load destinations.

Some Sacrifices

Fault isolation difficulty: A bug in one destination can crash the entire monolith, despite extensive automated testing.

Reduced in‑memory cache efficiency: Previously, low‑traffic destinations ran in few processes with hot caches; now cache is spread across thousands of processes, lowering hit rates.

Dependency updates affect many destinations: Updating a shared library version can break multiple destinations, though automated tests help detect differences.

Conclusion

Introducing microservices and isolating destinations solved pipeline performance issues, but lacking proper tools for bulk updates and deployments caused a rapid decline in developer productivity.

Architecture choices involve trade‑offs across multiple dimensions; there is no absolute best solution.

Key considerations include evaluating new complexity, its assessability, and mitigation strategies (e.g., shared‑library versioning).

Operational cost impact must be acceptable, especially regarding load‑pattern challenges.

While enjoying new architecture benefits, teams must retain control over scaling, tuning, and management overhead.