Evolution of Machine Management at Spotify: From ServerDb to Cloud Migration

Background : Spotify, founded in 2006, became the world’s largest licensed music streaming platform and by early 2016 operated roughly 12 000 physical and virtual servers. Rapid traffic growth around 2012‑2013 exposed the limitations of its manual operations processes.

Pre‑2012 – Early Ops : An Ops department managed servers manually, creating the first internal tool ServerDb – a simple SQL database with scripts that stored hardware details, location, network info, and status (e.g., ‘in use’, ‘broken’, ‘installing’). Supporting systems included moob for out‑of‑band management, DNS for service discovery, and Puppet for configuration.

Installation automation relied on tools such as Fully Automated Installer, later replaced by Cobbler, Debian‑installer, and eventually Spotify’s own Duck . Despite automation, many steps still required manual intervention, leading to unpredictable provisioning times and two major pain points: unstable installation processes and a slow, manual resource‑request workflow.

Key Positive Designs : DNS was used for service discovery via SRV/TXT records, and ServerDb acted as a trusted single source of truth for machine metadata.

Late 2013 – The Turning Point : The newly formed Infrastructure & Operations (IO) team created an Agile squad to address Ops problems. They adopted an MVP approach, focusing on three high‑impact areas: DNS changes, machine system registration, and resource‑request automation.

DNS Automation : Manual zone‑file edits were replaced by a pipeline that generated zone data from ServerDb, added integration tests and code reviews, and automatically pushed updates to the DNS master via a scheduled job.

Machine System Registration : ServerDb exposed a REST API; CSV‑based registrations were error‑prone. The goal shifted to fully automated registration using iPXE, which queried ServerDb to determine the appropriate boot environment (production OS, Duck‑provided installer, or unknown‑machine handling). The machine’s serial number became the unique identifier.

Resource‑Request Automation : The original provgun script fetched JIRA requests and triggered provisioning but lacked error handling. It was replaced by provcannon , a Python rewrite that added return‑code checks and retry logic, running twice daily to process batches.

The MVP effort dramatically reduced request processing from weeks to hours, eliminated daily DNS updates, and freed time for further improvements.

2014 – Scaling the Solution : With faster provisioning, new engineers sought even quicker turn‑around. Additional automation addressed machine reboot and recycle tasks, leading to the development of two key services:

Neep : A lightweight REST service built with Pyramid and RQ that handles install, recycle, and reboot actions across data centers, leveraging ServerDb state changes to drive iPXE boot flows. Example job output:

{
  "status": "finished",
  "result": null,
  "params": {},
  "target": "hardwarename=C0MPUT3",
  "requester": "negz",
  "action": "install",
  "ended_at": "2015-07-31 17:45:53",
  "created_at": "2015-07-31 17:36:31",
  "id": "13fd7feb-69d7-4a25-821d-9520518a31d6",
  "user": "negz"
}

Sid : The primary UI for resource requests, also a Pyramid REST service, which queries ServerDb for inventory, fetches role data, and invokes Neep to perform actions. Sid’s UI, built on the Lingon framework, eventually replaced JIRA as the preferred request portal.

Additional improvements included rewriting DNS zone generation and using pre‑built system images for faster installations. These changes cut request‑to‑completion times from hours to minutes, making Sid the most popular resource‑management tool at Spotify.

Post‑2015 – Cloud Migration : In 2015 Spotify decided to adopt public cloud, selecting Google Cloud Platform (GCE). To guide engineers toward optimal configurations, the Spotify Pool Manager (SPM) was created as a thin adaptation layer over the GCE API, exposing sensible defaults and managing instance groups. SPM is a stateless Pyramid service that orchestrates both GCE resources and existing physical machines via Sid, providing a unified provisioning experience.

The overall evolution demonstrates how business growth drives operational transformation: from manual processes to automated provisioning, from on‑premise servers to a hybrid cloud model, all built around a trusted data source (ServerDb) and service‑oriented architecture.