Lessons from Google, eBay, and Amazon on Large‑Scale Multi‑Language Microservice Architecture

Multi‑Language Microservices Are the End Game

When you look at large‑scale systems from Google, Twitter, eBay and Amazon, their architectures have converged to a similar shape: a collection of multi‑language microservices.

Randy Shoup, who held senior positions at Google and eBay, explores this idea in his talk “Scale‑Service Architecture: Lessons from Google and eBay.” He immerses the audience in experiences they may not have had: creating, using, evolving, and protecting massive architectures.

He asks what a large‑scale multi‑language microservice ecosystem looks like, how it feels to operate such services, what it looks like from a service‑owner perspective, and what can go wrong.

The key insight is the deliberate adjustment of incentive mechanisms, which underlies the whole effort. Small teams build clean services, internal refund models make services self‑sustaining, architecture evolves without dedicated architects, and clean design is motivated bottom‑up.

Conscious incentive alignment helps scale large, dynamic organizations and codebases. Removing locks, sharing no state, communicating via messages, and parallelising work let distributed systems accomplish more.

Multi‑Language Microservices Are the Final Game

Large‑scale systems ultimately evolve into a set of multi‑language microservices. “Multi‑language” means services can be written in many languages.

eBay’s evolution (5th‑generation architecture):

Started in 1995 as a single‑file Perl app.

Moved to a single‑file C++ app, eventually a DLL with 3.4 million lines of code.

Experience pushed toward a more distributed Java‑partitioned system.

Today eBay runs many Java services plus a suite of multi‑language microservices.

Twitter’s evolution (3rd‑generation architecture):

Started as a monolithic Ruby on Rails app.

Shifted to a front‑end JavaScript + Rails combo, with many Scala services on the back‑end.

Eventually transitioned to today’s multi‑language microservice set.

Amazon’s evolution:

Started as a single‑file C++ app.

Moved to services written in Java and Scala.

Ended with a collection of multi‑language microservices.

Service Ecosystem

What does a large‑scale ecosystem of multi‑language microservices look like?

At eBay and Google, hundreds to thousands of independent services work together.

Modern large systems compose services as a graph rather than a strict hierarchy.

Services depend on many others while being depended on by many.

Older large systems were organized in rigid layers.

How to Build a Service Ecosystem?

Best‑performing systems are evolutionary products, not top‑down designs. For example, Google never had a top‑down design; it grew organically over time.

Variation and natural selection. New services are created to solve problems, or are extracted from existing services. Services persist as long as they provide value; otherwise they are retired.

Bottom‑up development. Clean design can emerge as a new product rather than being imposed from above.

Example: Google App Engine’s service layers – Cloud Datastore built on Megastore, which is built on Bigtable, which rests on Colossus, which runs on Borg – illustrate a clean, bottom‑up stack.

Google operates without a formal “architect” title; technical decisions are made locally by teams, not centrally approved.

eBay, by contrast, still has an architecture review board that must approve large projects, creating bottlenecks.

eBay improves this by encoding expert knowledge into reusable libraries, services, or guidelines that teams can adopt without waiting for a review.

How Standards Evolve Without Architects

Decentralized control can still lead to standardization.

Standardization often occurs in communication between services and shared infrastructure.

Standards survive because they are healthier than alternatives.

Typical standardized communication layers:

Network protocols – Google uses a proprietary protocol called Stubby; eBay uses REST.

Data formats – Google uses Protocol Buffers; eBay leans toward JSON.

Interface definition standards – Protocol Buffers for Google, JSON schema for JSON.

Common standardized infrastructure includes source‑code control, configuration management, cluster managers, monitoring, alerting, diagnostics, etc.

In an evolutionary environment, standards are propagated through code, incentives, code reviews, and code search.

Creating New Services

New services are created once the idea is validated.

Often a feature is built for a specific use case, then discovered to be generally useful.

A small team forms and the service is split into its own unit.

This happens only when the functionality succeeds and fits many use cases.

Architecture grows pragmatically; no one dictates that a service must be added.

Examples: Google File System, Bigtable, Megastore, Google App Engine, Gmail – all started as internal projects that later became widely used.

Deprecating Old Services

When a service is no longer used, its technology can be recycled, people can be reassigned, and useful components may live on in other projects (e.g., Wave’s collaborative editing in Google Docs).

Building a Service

When you are the service owner in a large multi‑language microservice system, a well‑behaved service is single‑purpose, has a simple clear interface, is modular and independent, and does not share a persistent layer.

Goals of a Service Owner

Deliver needed functionality at appropriate quality, performance, stability, and reliability while continuously improving.

Do so with minimal cost and effort.

Align incentives to encourage reuse of common infrastructure.

Leverage battle‑tested tools, processes, components, and services.

Automate build and deployment.

Optimize resource utilization.

Responsibilities of a Service Owner

You build it and run it.

The team (typically 3‑5 engineers) handles design, development, deployment, and retirement.

No separate maintenance team.

The team freely chooses technology, methods, and environment and is accountable for those choices.

Services have bounded scope.

Cognitive load of the team is limited; they need not understand every other service.

They must deeply understand their own service and its dependencies.

Small teams (3‑5 people) enable high‑bandwidth communication and benefit from Conway’s Law.

Relationships Between Services

Think of service relationships as supplier‑customer, even within the same company.

Clear ownership and responsibility are essential.

Define SLAs; customers pay for services, aligning economic incentives.

Operating Services at Scale

Predictable performance is a requirement; tail latency dominates overall latency, and rare low‑level events can cause large spikes.

Resilience and Incremental Deployment

Use canary releases, feature flags, and staged rollouts to mitigate risk; roll back quickly if problems appear.

Monitoring and alerting must be abundant; you can never have too many alerts.

Service Anti‑Patterns

Large services. Too many services create a tangled dependency graph that is hard to reason about and scale.

Shared persistence. A common database layer re‑introduces coupling; microservices should keep storage isolated.