How Liulishuo Accelerated AI‑Powered English Learning with Alibaba Cloud EMR Serverless Spark

Background Introduction

Liulishuo is a technology‑driven education company that has independently developed advanced English speaking assessment, writing scoring engines, and a deep adaptive learning system, offering comprehensive solutions to improve users' listening, speaking, reading, and writing skills.

The business features include AI scoring for speaking and writing, personalized course recommendations based on user goals and ratings, data‑driven optimization of recommendation strategies, and data operations that enhance user engagement and satisfaction.

Original Architecture Pain Points

Elastic resource management issues: inflexible allocation, peak‑time task waiting, low utilization during off‑peak.

Cost problems: Master and Core nodes remain active and incur charges even when idle.

Performance bottlenecks: lack of acceleration technologies like Fusion, leading to slow execution on large datasets.

Operational complexity: multi‑component architecture makes fault diagnosis and recovery difficult.

Monitoring gaps: only cluster‑level monitoring, missing Spark task and resource metrics.

Scaling delays: manual, slow response to temporary capacity demands.

Why Choose Alibaba Cloud Serverless Spark

EMR Serverless Spark is a high‑performance Lakehouse product for Data+AI, offering a one‑stop data platform with development, debugging, scheduling, and operations capabilities. It is 100% compatible with the open‑source Spark ecosystem and integrates seamlessly with existing data platforms.

Metadata management: supports external Hive MetaStore integration.

Scheduling engine support: integrates mainstream schedulers such as Airflow and DolphinScheduler.

Comprehensive monitoring and alerting for real‑time task status and performance tracking.

Managed elastic scaling: automatically adjusts resources, reducing manual intervention.

Cluster stability: high reliability managed by the cloud provider.

Elastic resource management: on‑demand allocation avoids waste.

Pay‑as‑you‑go: billing only for actual resource usage, lowering costs.

Fast startup: no pre‑configuration needed, enabling rapid task execution.

Automatic scaling based on workload.

Performance optimization: Fusion acceleration and built‑in Celeborn service improve shuffle performance and reduce costs.

Technical Solution Design

The Liulishuo data platform covers the full lifecycle from data collection, ingestion, storage, computation, management, query, to visualization. It supports multiple data sources, provides millisecond‑level real‑time processing, minute‑level near‑real‑time streaming, and T+1 offline batch processing. ETL scripts are stored in GitLab with branch and version control, enabling traceable and collaborative development. Airflow handles workflow scheduling, while EMR Serverless Spark serves as the core compute engine, enhanced by the Fusion accelerator for efficient, elastic, reliable, and low‑cost processing. Hive Metastore offers a unified data catalog, OSS provides durable object storage, and AirEye with GoAlert delivers comprehensive observability and alerting.

Typical Application Scenarios

CI&CD and Offline ETL

Liulishuo’s DAG auto‑generation service triggers CI checks on submitted scripts, merges them, and creates Airflow DAG files that directly submit tasks to EMR Serverless Spark via Alibaba Cloud operators. This replaces the previous Airflow + EMR Gateway approach, delivering higher execution efficiency, concurrency, stability, and reliability, while reducing operational costs.

Data Integration

The platform supports batch and incremental ingestion from business databases, using Hudi for lake storage to handle updates, deletions, and schema evolution. CDC monitors database logs, pushes change events to Kafka, and schedules EMR Serverless Spark jobs for incremental lake loading, leveraging serverless elasticity for optimal resource utilization.

Data Query

Three query engines—Trino, Doris, and Spark—are available. Migrating the Spark engine to EMR Serverless Spark replaced a fragile Thrift Server setup, improving stability, enabling fine‑grained resource isolation, and reducing operational overhead. User‑level routing prevents resource contention, and elastic scaling cuts idle compute costs.

Benefits After Migration

Performance: Fusion‑accelerated offline tasks reduce execution time by 40%, delivering core reports earlier.

Stability: Task failure rate drops by 80%.

Resource flexibility: Automatic scaling matches business demand.

Operational cost: Simplified architecture lowers platform maintenance expenses.

Cost‑effectiveness: True pay‑as‑you‑go model reduces expenses by 30%.

Future Expectations

Leveraging Alibaba Cloud EMR Serverless Spark, Liulishuo built a fully managed, high‑performance offline data processing platform with the Fusion engine, vectorized computation, and RSS capabilities, achieving over threefold performance gains over open‑source versions. The rich feature set boosts team productivity and accelerates data‑driven product delivery. Liulishuo looks forward to collaborating with Alibaba Cloud to deliver more industry‑specific lakehouse solutions.