Stormcrow: Dropbox’s Scalable Feature‑Flag Platform for Rapid Deployment and A/B Testing
The article describes Dropbox’s Stormcrow system, a configurable feature‑gate platform that enables fast, safe rollout of new functionality across web, desktop, and mobile clients, supports granular A/B testing, leverages custom data fields, and integrates deployment, monitoring, and audit tooling for large‑scale operations.
Large internet companies typically roll out new features to a small subset of users for gray‑scale and A/B testing; Dropbox has introduced a similar platform called Stormcrow that lets engineers edit and deploy feature gates quickly and safely across the entire product stack.
Example
Stormcrow can be used to run an A/B test on German users, showing a red button to 33% of them, a blue button to another 33%, and no button to the remaining 34%, while English‑language users always see the blue button.
How to Define Populations?
A population is built from selector objects (e.g., user , session ) and datafield definitions that extract boolean, date, numeric, set, or string values and combine them with simple rule logic.
Example data field definition:
@dataField decorator specifies that the field requires a USER object and produces a STRING containing the user’s email.
Hive‑Based Population Classification: Connecting to Our Analytics Warehouse
When a population needs data that only exists in Dropbox’s Hive‑based analytics warehouse, a daily pipeline exports those Hive‑derived groups into a production‑readable store, trading freshness for scalability.
Derived Populations: Building Complex Groups from Simple Definitions
Derived populations allow combining basic groups, such as matching Android devices with a feature flag recents_web_comments set to OFF, reducing duplicated matching logic.
Selector Inference: Making the API Easier to Use
Stormcrow models Dropbox’s internal entities (e.g., user , team , identity ) as a graph. When a selector is provided, Stormcrow computes its transitive closure to infer additional selectors, caching the work in thunks to avoid unnecessary I/O.
Deployment: Web and Internal Infrastructure
Configuration data is stored in a JSON file stormcrow_config.json that is pushed to all production servers via Dropbox’s internal push system. A background thread called the “Stormcrow loader” watches the file and reloads it without restarting the server.
Deployment: Desktop and Mobile Clients
Clients fetch the configuration in bulk; mobile clients also send a selector describing the app and device, while desktop clients send a selector with host information, allowing platform‑specific data fields.
Monitoring
Every assignment and exposure of a gated feature is recorded in Dropbox’s real‑time monitoring system Vortex. The Stormcrow UI embeds charts that show per‑variant exposure over time, with vertical lines marking configuration changes.
Performance Risks
Because data fields can execute arbitrary logic, a poorly written field may become a performance bottleneck. The recommendation is to avoid any database or I/O calls inside feature‑gate evaluation and to pass as much context as possible from the caller.
Audit Challenges
Feature gates are not version‑controlled, so Dropbox provides a full audit history and a static analysis service that scans the codebase for Stormcrow usage, listing current occurrences and historical commit views. The service also notifies owners of stale or deprecated flags.
Quality Assurance and Testing
Stormcrow supports manual “override” of populations and data fields for testing, and a mocked Stormcrow implementation returns default variants for unit tests.
Conclusion
Building a unified feature‑gate service at Dropbox’s scale required careful design across infrastructure, data ingestion, configuration management, UI, and tooling, and the lessons shared here can help teams building their own flag systems.
Why is the new platform called Stormcrow? It replaces the old system named Gandalf, and “Stormcrow” is one of Gandalf’s names in Tolkien lore.
High Availability Architecture
Official account for High Availability Architecture.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.