Silicon Valley's Data Middle Platform Secrets: EA, Twitter, Airbnb, Uber

1. Silicon Valley's Middle Platform Theory

Although the term "data middle platform" originated in China, Silicon Valley companies have been building equivalent data platforms for years, often called "data platforms". These platforms provide unified data access, abstraction, sharing, and reuse across business units, enabling data‑driven product development.

2. EA's Data Platform Construction

2.1 EA's Game Portfolio

EA organizes its games into several categories:

Sports (e.g., FIFA, Madden, NBA)

Shooter (e.g., Battlefront)

Social (e.g., The Sims 4)

Mobile (e.g., Plants vs. Zombies, RealRacing)

2.2 EA's Game Data

Since 2012 EA's big‑data department collects player information from Facebook or email, uses push notifications for acquisition, and tracks in‑game advertising, banner personalization, and overall player experience.

Example: In FY2019 FIFA had 4.5 million unique players, generating nearly 500 k matches and 3 million shots within 90 minutes.

2.3 EA's Platform Evolution

Early analysis stage : Each studio maintained its own siloed analytics pipeline, causing multi‑day data latency.

In 2012 EA created a digital platform department to unify data collection, reducing latency to a few hours.

Data standardization : Established a company‑wide taxonomy covering player consumption, social behavior, and gameplay metrics (e.g., average session length, retention, channel analysis).

Data specification : Defined two telemetry event attribute groups—common attributes (player ID, device, game name, event ID, timestamp) and special attributes specific to business needs.

2.4 Data Platform Architecture

Data flows from client devices (mobile, console, PC) into a telemetry layer, then into the capture layer ( River) where two collectors operate:

Lightning – real‑time ingestion.

Tide – batch ingestion.

Collected data is stored in Ocean , a hybrid storage system using HDFS for hot data and AWS S3 for cold data.

ETL processing is handled by Shark (with custom Onzie workflow manager). Processed data is written to two warehouses:

Pond – a real‑time warehouse built on Couchbase, supporting self‑service queries and feeding hundreds of daily jobs.

Pearl – a traditional warehouse originally on Hadoop Tide, later migrated to AWS Redshift for BI tools.

2.5 Capability Reuse

EA built reusable services such as a tagging system for FIFA that quickly identifies target players for promotions, an anti‑fraud model to detect illicit in‑game currency sales, and AB‑testing pipelines that feed results back into the game.

2.6 Summary of EA's Practices

Quarterly release of new platform features.

Self‑service analysis tools for business units.

Abstracted capabilities for cross‑team reuse.

Closed‑loop feedback from data insights to product improvements.

3. Data Platforms of Other Silicon Valley Unicorns

3.1 Twitter

Twitter's pipeline mirrors EA's: production logs are ingested via Kafka (real‑time) or Gizzard (batch) into a Hadoop ecosystem. Data is exposed through DAL (Data Access Layer) for analysts, and stored in MySQL, Vertica, and a custom key‑value store Manhattan . Real‑time processing uses Storm/Heron, while batch jobs run on Hadoop.

3.2 Airbnb

Airbnb collects event logs and MySQL dumps via Sqoop, streams them through Kafka to a Gold Hive cluster, replicates to a Silver Hive cluster for self‑service queries, and finally processes data on Spark with storage in S3. Presto, Airpal, and Tableau provide ad‑hoc querying and visualization.

Use cases include image‑based recommendation, sentiment analysis of reviews, dynamic pricing models, and collaborative filtering for host preferences.

3.3 Uber

Uber ingests data from micro‑services, MySQL, schemaless stores, and Cassandra. Cassandra feeds high‑frequency data into Kafka, then into the Marmaray collector, which writes to HDFS using Hudi for incremental processing. Data is served to analysts via Vertica.

Key capabilities include dynamic pricing models, driver demand prediction, city‑wide real‑time visualizations, and data‑driven autonomous‑driving simulations.

4. Q&A Highlights

Q1: How are telemetry metrics defined? Metrics must cover all user actions and support business analysis (DAU, MAU, average session length, spend, etc.) with a standardized format.

Q2: Why do some data platform projects fail? Failure often stems from lack of phased planning, insufficient information‑technology foundation, and neglecting data governance and standardization.

Q3: Are big‑data platforms the same as data middle platforms? Big‑data platforms focus on storage and processing, while a data middle platform adds abstraction, sharing, and reuse of data capabilities across business lines.

Q4: What metrics evaluate platform investment? ROI, hardware and personnel costs, resource consumption per department, and impact on core business KPIs.

Q5: Is the data middle platform market mature in China? Many Chinese firms are now at the data‑platform stage, moving from siloed analytics to unified, reusable data services.