How Spotify Scaled Machine Management: From Ops Chaos to Cloud Automation

Spotify Overview

Spotify, founded in Sweden in 2006, launched its streaming service in 2008 and grew to become the world’s largest licensed music‑streaming platform. By Q1 2016 the company operated roughly 12 000 physical and virtual servers.

Pre‑2012: The Dark Age

In its early years the Ops team handled all operations manually, often firefighting. To avoid hand‑crafting server inventories they built the first internal tool, ServerDb , a simple SQL database plus scripts that stored hardware specs, rack locations, network interfaces, and a unique hardware identifier. ServerDb also tracked server status such as ‘in use’, ‘broken’, or ‘installing’.

Supporting systems included:

moob – out‑of‑band (OOB) management

DNS – service discovery and configuration

Puppet – configuration management

Installation was initially handled by the Fully Automated Installer, later replaced by Cobbler and Debian‑installer, and eventually by Spotify’s own Duck tool. Despite automation, many steps still required manual JIRA requests and weeks‑long provisioning cycles.

Unstable server installation processes required frequent rewrites. Low automation of resource request workflows caused long delivery times.

DNS provided reliable service discovery via SRV and simple load‑balancing via TXT records. ServerDb acted as a single source of truth for machine configuration and status.

End of 2013: The Turning Point

The newly formed Infrastructure & Operations (IO) team took over Ops and created an Agile squad to tackle operational pain points. Their MVP focus targeted three areas: DNS updates, machine inventory ingestion, and resource‑request automation.

DNS Automation

Manual zone‑file edits were replaced by a pipeline that generated zone data from ServerDb, added integration tests and code reviews, and automatically pushed changes to the DNS master via a scheduled job.

Machine Inventory Ingestion

ServerDb exposed a REST API; machines were initially imported from CSV files, which were error‑prone. The goal became fully automated, hands‑free registration using iPXE to boot machines into environments based on ServerDb status (e.g., ‘installing’ boots into Duck’s installer).

Resource‑Request Automation

The original provgun script fetched JIRA requests and triggered ServerDb updates, but lacked error handling. It was superseded by provcannon, a Python rewrite with return‑code checks and retry logic, running twice daily to process batches.

Applying the MVP principle reduced resource‑request handling from weeks to hours, eliminated manual DNS updates, and freed time for further system improvements.

2014: Catch‑Up

After achieving hour‑level provisioning, new engineers accustomed to public‑cloud speeds demanded faster cycles. Remaining manual steps (reboots, decommissioning) prompted the development of a self‑service platform and API.

Neep

Neep is a lightweight REST service built with Pyramid and RQ that handles installation, recycling, and rebooting of machines across data centers via OOB management. An example Neep job:

{
  "status": "finished",
  "result": null,
  "params": {},
  "target": "hardwarename=C0MPUT3",
  "requester": "negz",
  "action": "install",
  "ended_at": "2015-07-31 17:45:53",
  "created_at": "2015-07-31 17:36:31",
  "id": "13fd7feb-69d7-4a25-821d-9520518a31d6",
  "user": "negz"
}

Sid

Sid is the primary UI for engineers to request resources and manage machines. It queries ServerDb for inventory, fetches role data from internal micro‑services, and delegates actions to Neep. Sid’s UI, built on Lingon, has replaced JIRA as the preferred request portal.

These tools, together with DNS zone‑generation rewrites and automated image creation, cut provisioning and reclamation times from hours to minutes, making Sid the most popular resource‑management tool at Spotify.

Post‑2015: Moving Toward the Light

In 2015 Spotify began migrating to Google Cloud Platform (GCE), motivated primarily by cost savings and more efficient resource allocation. To simplify GCE usage, the Spotify Pool Manager (SPM) was created as a stateless Pyramid service that wraps the GCE API, offering sensible defaults and managing instance groups.

SPM integrates with Sid, allowing engineers to request pools that span both GCE instances and remaining on‑premise servers, preserving a unified workflow.

Conclusion

Spotify’s journey—from manual Ops to automated, service‑oriented infrastructure and hybrid cloud management—illustrates a typical evolution for fast‑growing tech companies. By establishing a trusted data source (ServerDb), automating DNS, provisioning, and adopting self‑service platforms, they reduced request latency from weeks to minutes and built a scalable foundation capable of handling tens of thousands of machines.