How Asana Scaled Its Data Infrastructure: From MySQL to Redshift & Beyond

Background and Initial Architecture

Asana’s early data platform consisted of a handful of Python scripts and a single MySQL instance running on one server. This simple stack was sufficient while the user base was small, but it quickly became a bottleneck as the company grew.

Key Limitations

Long‑running aggregation queries on MySQL.

Fragile log‑processing pipelines that required manual intervention.

Insufficient monitoring and testing, leading to frequent firefighting.

Infrastructure Improvements

Monitoring, Testing, and Automation

Invested in systematic monitoring (CPU, memory, Redshift load), automated alerts, and a unit‑test suite for data‑pipeline code. This reduced manual debugging and gave early warning of resource exhaustion or data‑quality anomalies.

Orchestration with Luigi

Replaced ad‑hoc cron jobs with Luigi pipelines. Luigi tracks task dependencies, aborts downstream jobs on failure, and provides automatic alerts. Incomplete runs can be resumed without re‑executing successful tasks, dramatically cutting manual clean‑up time.

Migration to Amazon Redshift

MySQL’s row‑store architecture could not handle the growing volume of analytical queries. After building a custom histogram‑based cache with limited success, the team migrated the analytical workload to Amazon Redshift. The same SQL statements that took up to six hours on MySQL executed in a few seconds on Redshift, with no code changes.

Migration Process

Abstracted data‑load logic to use Amazon S3 as the staging area.

Implemented bulk COPY commands to load data from S3 into Redshift.

Temporarily wrote to both MySQL and Redshift during the transition to avoid service disruption.

Coordinated cross‑team effort to migrate inter‑dependent queries in the correct order.

Scalable Log Processing with Amazon EMR

Log volumes grew beyond what a single MySQL instance could ingest. The team adopted Hadoop MapReduce via the Python mrjob library and Amazon Elastic MapReduce (EMR). The workflow:

Raw logs are written to S3.

mrjob jobs run on an eight‑node EMR cluster, performing transformations and aggregations.

Processed results are loaded directly into Redshift.

This architecture yields a 4–6× speedup over the original scripts and keeps end‑to‑end latency under 24 hours.

Business Intelligence Tools

Two self‑service BI platforms were evaluated:

Interana – provides ultra‑fast raw‑log exploration and interactive funnel analysis.

Looker – connects to Redshift and delivers near‑real‑time query performance, allowing analysts without SQL expertise to build dashboards, cohort analyses, and anomaly investigations.

Future Directions

Deploy Hive on top of Redshift or a separate data lake for more flexible SQL querying.

Explore streaming analytics platforms for near‑real‑time ingestion.

Evaluate faster Hadoop alternatives such as Apache Spark for in‑memory MapReduce workloads.

Improve automated anomaly detection and trend‑based alerting.

Further reduce single points of failure by increasing redundancy in monitoring and orchestration.