Why Segment Merged 140+ Microservices: The Hidden Costs of Service Granularity
Segment consolidated over 140 microservices into a monolith to reduce change fan-out and operational burden, while Prime Video moved video frame processing in-memory to eliminate 14.4 million hourly object storage handoffs; both cases show service boundaries must justify their cost in coordination, data transfer, and failure isolation.
Segment's context: 140+ services, three engineers, maintenance burden
Segment's destination forwarding system received customer events and routed them to external APIs. With over 140 services, three full‑time engineers spent most of their time keeping the system running, leaving little capacity for product work. The team merged these services back into a monolith. The guiding principle: shared changes and frequent handoffs deserve scrutiny first; independent deployment that isn't actually used should be merged. Remote boundaries must prove their value through actual independence gained.
After splitting, changes fork
Early on, a shared queue caused head‑of‑line blocking: a slow destination would stall deliveries for all others. Splitting by destination — separate queues, separate services — gave clear fault isolation: external API failures were contained.
Maintenance introduced a new problem. Each destination had its own code repository but shared a set of common libraries. When a library was fixed, every service had to upgrade, test, and deploy separately. If only the service needing the fix was upgraded, the shared logic gradually diverged across services.
This is change fan‑out : one logical modification must be applied to multiple deployment units. Splitting gave the ability to deploy independently, but it also forced the team to synchronize upgrades. If the services were truly owned by different teams, independent deployment would offset this overhead. When the same people modify, test, and on‑call for all services, repository count grows without a corresponding increase in autonomy. The cost appears as delays, compatibility patches, and reluctance to touch shared libraries.
Operational account: scaling and testing
Different destinations had different load profiles, so scaling configuration had to be managed per service. After merging, a shared processing pool absorbs traffic spikes more easily.
For regression testing the team used a yakbak -based Traffic Recorder. It records HTTP requests and responses during an initial test run, then replays the recorded files in place of external APIs. The original transformation logic runs against the replayed responses. If a shared dependency breaks the transformation, the regression suite catches it; external API compatibility still requires live integration tests.
Merged logic, external scheduler: Centrifuge
Putting all events back into a single FIFO queue would bring back the original head‑of‑line blocking. Segment built Centrifuge to handle scheduling and retries. It organizes virtual queues by (data source, destination) pairs and uses database queries to adjust execution order. Failed tasks are pushed back; healthy tasks proceed first, without copying the entire event payload.
In Calvin French‑Owen's public Centrifuge design, the Director receives and executes jobs. It first writes the event and delivery info to an RDS MySQL JobDB (payload in jobs, state transitions in job_state_transitions, next retry time in retry_at). Scheduling only changes execution order; the payload stays in place.
The Director sends an HTTP request to the job's endpoint. A 200 records success; timeout or 5xx triggers a retry with a new retry_at based on the task's backoff policy. The Director confirms the endpoint responded — business side‑effects are the receiver's responsibility. On the expiry path, waiting tasks are archived to S3 and then marked complete.
On restart, the Director reads pending tasks from JobDB and resumes processing. A Consul session lock ensures a single writer per database. The team still manages task state, scheduling fairness, and backlog recovery. Segment eliminated the distributed runtime burden but kept the work of reliable delivery.
Two failure types must be distinguished. Downstream slowness or rate‑limiting can be isolated at the scheduler level. Code defects that crash the shared process will affect all co‑located tasks. Segment explicitly accepted this regression in isolation.
During migration, if the downstream supports idempotency keys, retry with the same key. If it only supports result queries, check the result: if completed, stop; if not found, pause automatic retry for manual verification. Resume only after confirming the original request terminated without effect.
Prime Video saved per‑frame handoffs
Prime Video's 2023 audio/video quality monitoring redesign revealed another fan‑out: the same intermediate data had to be ferried across multiple components.
In the original design (publicly described by Marcin Kolny), a media converter extracted video frames, wrote each frame to S3, detectors read the frames back for analysis, and Step Functions orchestrated the workflow — multiple state transitions per second of video.
At scale, the handoffs themselves became a scaling and cost bottleneck. The team moved conversion, detection, and orchestration into a single process, passing intermediate data via memory, deployed on EC2 and ECS. Module responsibilities were largely preserved; remote data passing and remote orchestration were removed. Each frame avoided one object write and subsequent read, plus the corresponding orchestration overhead.
When a single instance couldn't accommodate more detectors, the team replicated the service by detector subset and added an external request distributor. The final shape: multiple coarse‑grained processing units with tight internal collaboration and continued external division of labor.
The two companies cut costs at different points. Segment reduced maintenance work stemming from shared changes; Prime Video reduced storage and orchestration work stemming from data handoffs. Both separated business responsibilities from deployment boundaries.
Using 14.4M handoffs to derive a boundary
Illustrative calculation: 100 video streams, 10 frames/second, 0.2 MB/frame, 3 detectors. Frames per hour: 100 × 10 × 3600 = 3.6M. Each frame: 1 write + 3 reads = 4 object requests → 3.6M × 4 = 14.4M requests/hour. Data volume: 0.72 TB write, 2.16 TB read, total 2.88 TB logical I/O. These numbers represent inter‑component handoffs — not public traffic bills, not compute for detection algorithms. In‑process sharing eliminates the object‑service round‑trip; detectors still read memory.
If frame rate drops to 1 fps, requests fall 10× to 1.44M/hour, data to 0.288 TB. The remote boundary burden shrinks tenfold, making independent scaling or independent deployment more likely to pay off.
Another scenario: only one detector's demand grows sharply. Scaling the whole unit replicates the converter and the other detectors unnecessarily. Split hot detector A from stable B and C. A's replicas share the full frame stream (each frame read by one replica); B and C stay in‑process. Object storage: 1 write + 1 read per frame = 7.2M requests/hour, 1.44 TB logical I/O. Compare this handoff cost against the extra compute from scaling the entire unit.
Therefore, the key inputs for granularity are processing frequency, intermediate data size, consumer count, and growth disparity among components. Lines of code have almost no explanatory power for these costs.
One public protocol upgrade pulls how many services
For domestic channel integration projects, use a public protocol upgrade as a probe: do multiple adapters always change together because of the same field change? If external interfaces are controlled by different organizations with independent release windows and credentials, those boundaries have retention value. For fixed file‑processing chains, check how many steps repeatedly write intermediate files to disk.
Implementation steps:
Pick a set of co‑changing modules.
Observe a protocol upgrade: does it always touch the same services?
Observe a scaling event: are they always scaled together?
Observe failures: are they always investigated by the same team?
If answers are consistently yes, list them as merge candidates, then filter by fault and permission boundaries.
Scheduling keys must align with actual rate‑limiting objects. If an external API limits by customer credential, all data sources under that credential should share the quota. Splitting only by source would cause aggregate over‑limit. Code colocation and post‑failure task progression are two separate design decisions.
Preserve permission and fault boundaries centrally. Components holding different sensitive credentials, high‑risk processors that must be crash‑isolated, compute modules with divergent load growth, and teams with genuine independent deployment needs should stay separate. If the merged process acquires all permissions, the original access controls must be redesigned.
Validation does not require a full‑system rollout. Test the candidate group: compare old vs. new business results, latency, and resource usage with identical input; simulate a downstream timeout; verify other tenants continue unaffected; confirm backlog drains after recovery; ensure retries do not cause duplicate business actions.
Domestic teams should also factor in human time: for a single shared‑library upgrade, record how many repositories changed, how many integration rounds occurred, and how many on‑call hours were consumed. Cloud costs appear on the bill; change fan‑out is deducted from the delivery cycle.
Every retained service must be able to state who it protects and who pays the maintenance and recovery costs.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
