Lessons from Google, eBay, Twitter, and Amazon on Scaling Multi‑Language Microservices

Multi‑Language Microservices Are the End Game

Large‑scale systems at Google, Twitter, eBay and Amazon have all evolved into a similar architecture: a collection of microservices written in many programming languages.

Randy Shoup, who held senior roles at Google and eBay, explores this idea in his talk “Scaling Service Architecture: Lessons from Google and eBay,” immersing the audience in the creation, use, evolution, and protection of massive architectures.

He asks what a massive multi‑language microservice ecosystem looks like, how it feels to operate such services, what it is like to be a service owner, and what can go wrong.

A key insight is the conscious adjustment of incentive mechanisms, which explains why small teams build clean services, why internal service refund models are powerful, how architecture can evolve without dedicated architects, and how standards emerge without a central committee.

Incentives, when aligned, drive the growth of large, dynamic organizations and codebases; removing locks, sharing no state, communicating via messages, and parallelising everything lets more work get done in a distributed system.

Evolution of Multi‑Language Microservices

eBay started in 1995 with a single‑file Perl app, moved to a C++ DLL with 3.4 million lines, then to a more distributed Java partitioned system, and today runs a large set of multi‑language microservices.

Twitter began as a monolithic Ruby on Rails app, shifted to a front‑end JavaScript/Rails combo with heavy Scala back‑end, and now also operates a multi‑language microservice stack.

Amazon followed a similar path: from a single C++ app to services written in Java and Scala, ending with a multi‑language microservice architecture.

Service Ecosystem

In a large multi‑language microservice ecosystem, hundreds to thousands of independent services collaborate, forming a graph rather than a strict hierarchy.

Modern large systems compose services as a relationship graph, not a layered hierarchy.

Services depend on many others while being depended upon by many.

Older large systems were organized in rigid layers.

How to Build a Service Ecosystem

These high‑performing systems are evolutionary products, not top‑down designs. At Google, for example, the architecture grew organically over time.

When a problem arises, new services are created, or existing services are repurposed. Services persist as long as they provide value; otherwise they are retired.

Clean design emerges as a by‑product of useful services rather than from a top‑down blueprint.

Google’s stack illustrates this evolution: Cloud Datastore built on Megastore, which sits on Bigtable, which relies on Colossus, which runs on Borg.

Each layer adds new capabilities not present in the layer below, and the stack was built bottom‑up, not top‑down.

Google operates without formal architects; technical decisions are made locally by teams, not centrally approved.

eBay, by contrast, historically required an architecture review board, which became a bottleneck.

eBay’s improvement was to encode expert knowledge into reusable libraries, services, and guidelines, reducing reliance on late‑stage reviews.

How Standards Evolve Without Architects

Without central control, standards still emerge through common communication protocols, data formats, and interface conventions.

Network protocols (e.g., Google’s Stubby, eBay’s REST).

Data formats (e.g., Protocol Buffers, JSON).

Interface schema standards.

Common infrastructure components such as source control, configuration management, cluster managers, monitoring, alerting, and diagnostics also standardise through convention.

Standards propagate via code, incentives, code reviews, and searchable code bases.

Creating New Services

New services are created once the underlying idea is validated.

Often a feature built for a specific use case proves useful more broadly.

A small team owns the service from design through retirement.

The team freely chooses technology and is accountable for its choices.

Teams stay small (typically 3‑5 engineers), enabling high‑bandwidth communication and aligning with Conway’s Law.

Service Owner Goals and Responsibilities

Deliver needed functionality with appropriate quality, performance, stability, and continuous improvement.

Do so with minimal cost and effort.

Align incentives to encourage reuse of shared infrastructure.

Maintain full ownership of the service lifecycle.

Service Relationships

Treat service interactions as supplier‑customer relationships, even within the same organization, with clear ownership, SLAs, and economic incentives.

Operating Services at Scale

Predictable performance is a requirement; tail latency dominates overall latency, and rare low‑level events can cause large spikes.

Resilience requires handling machine, cluster, and data‑center failures, load‑balancing, traffic‑shaping, and rapid roll‑backs.

Incremental deployment using canary releases and feature flags allows safe roll‑out and quick rollback if issues arise.

Monitoring must be abundant; you can never have too many alerts.

Service Anti‑Patterns

Too many services lead to a bloated ecosystem that is hard to reason about, scale, and change.

Sharing persistence layers (e.g., a common database) re‑introduces coupling and undermines microservice isolation.