Real‑Time Computing for Online Education: Architecture, Data Platform and Automation at VIPKID

VIPKID is an online K‑12 English education platform that serves over 700,000 paying students across 63 countries, delivering more than 100,000 one‑to‑one lessons per day with peak concurrency of 35,000 lessons.

The core business scenario involves a teacher and a student interacting via live video, chat, and interactive whiteboard, requiring tight synchronization of audio, video, and UI events across multiple components.

Current pain points include delayed manual handling of Help requests (about 10% of sessions), low efficiency of human monitors, and hidden issues when users cannot reach staff.

Technical implementation focuses on a real‑time data pipeline built on Apache Flink:

Data platform (Vlink) provides unified data ingestion, business semantics, and point‑tagging rules.

Business data collection captures events from mobile, PC and backend services, especially the "enter classroom" flow.

Business data computation uses Flink streaming jobs for event‑time union, multi‑stream joins, and asynchronous dimension loading, with code examples such as: env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) val stream = env.addSource(singleSource).name("signal") .union(env.addSource(avSource).name("av")) .union(env.addSource(dbySource).name("dby")) .union(env.addSource(enterSource).name("enter")) .filter(_.f) .filter(_.unNature) .assignTimestampsAndWatermarks(new DummyEventTimePunctuWaterMarks[InlineInputEventForm](6000)) .filter(m => *** ).name("***") // further processing ... sink***ToKafka(...)

Support for various sinks (filtered data, unnatural data, late data) and sources (Kafka, HBase, Elasticsearch, JDBC).

The automation system consumes the real‑time tags to automatically resolve Help requests before human monitors intervene, applying validation, self‑check, and corrective actions, while falling back to manual handling for unresolvable cases.

Key challenges addressed include data quality across 11 teams, high‑throughput dimension lookups, and handling of "unnatural" timestamps; solutions involve unified metric management, windowed aggregation, caching, and batch job orchestration.

Results show a ~3% reduction in user Help requests, 60% of issues auto‑resolved, 40% fewer monitoring staff, sub‑20‑second resolution time, and a two‑third drop in complaint rate.

The work demonstrates that real‑time streaming is essential for online education services and that the built tag system can be reused for other scenarios such as full‑link fault detection and post‑class analytics.

References

[1] https://www.donews.com/news/detail/4/2978938.html

[2] https://blog.scottlogic.com/2018/07/06/comparing-streaming-frameworks-pt1.html

[3] https://issues.apache.org/jira/browse/FLINK-13148

[4] https://github.com/google/guava/wiki/CachesExplained

Author : Zhen Guoyou, Senior Data Engineer at VIPKID Online Classroom R&D Center, responsible for real‑time computing architecture and data platform construction.