How Asana Scaled Its Data Infrastructure: From MySQL to Redshift & Hadoop

Starting Small, Growing

Asana began with a simple system of Python scripts and a MySQL database on a single machine, which was sufficient for early development but soon hit performance limits as the company grew steadily since 2011.

Ending Endless Issues

Rapid growth exposed data robustness problems, causing frequent firefighting and unreliable metrics. The team adopted a "5 Whys" approach, improved logging, monitoring, and alerting to differentiate real issues from noise.

Inspired

Automated testing was introduced, moving from ad‑hoc cron jobs to Luigi pipelines that manage task dependencies, provide alerts on failures, and allow partial restarts, reducing manual cleanup.

Extended Data Warehouse (Redshift)

MySQL struggled with large‑scale analytical queries, prompting a migration to Amazon Redshift. Queries that took hours on MySQL now run in seconds on Redshift, dramatically improving performance and accessibility for non‑technical users.

Migration Process

The migration required abstracting Redshift features, handling S3 data loads, and coordinating cross‑team efforts to rewrite dependent MySQL queries, often running both databases in parallel during transition.

Unlocking New Analytics

With Redshift, the team evaluated BI tools, ultimately adopting Looker, which delivered near‑real‑time query performance and enabled business users without SQL expertise to explore data independently.

Further Expansion

Redshift’s resource‑limiting capabilities helped prevent single‑process monopolization, and scaling the cluster became a simple button‑click operation, with plans for automation.

Scalable Log Processing (Elastic MapReduce)

To handle growing log processing latency, Asana adopted Hadoop Elastic MapReduce using the Python‑based mrjob framework, achieving 4‑6× performance gains on an eight‑node cluster and simplifying data movement from Hadoop to Redshift.

Business Intelligence Tools (Interana and Looker)

Interana was introduced for fast raw‑log analysis, enabling sub‑second queries over billions of events, while Looker provided flexible visualizations for financial, revenue, and user‑behavior insights.

Next Steps

The team plans to explore adding Hive on top of Redshift, streaming analytics, faster Hadoop alternatives like Spark, improved anomaly detection, and reducing single points of failure.