1 views collected around this technical thread.
Shopee’s deployment of StarRocks across DataService, DataGo, and DataStudio demonstrates that its vectorized engine, cost‑based optimizer, and materialized‑view caching can query Hive, Iceberg, Delta Lake and Hudi up to 20,000× faster than Presto, cutting CPU usage and delivering consistently lower latency for complex analytics.
This article presents Shopee's calibration‑based meta‑learning approach for personalized click‑through‑rate prediction in live streaming, detailing business context, modeling goals, model evolution from Calibration4CVR to CBMR, EmbCB and MlpCB optimizations, and multi‑task and multi‑scene extensions that achieve significant AUC and business metric improvements.
ShopeeVideo OCR is a multi‑language text‑recognition system for Southeast Asian e‑commerce videos that unifies detection, Transformer‑based recognition, layout analysis, and large‑scale synthetic data generation to handle Indonesian, Filipino, English, Vietnamese, Thai and Chinese scripts, delivering industry‑leading accuracy and winning thirteen ICDAR first‑place awards.
Shopee faced fragmented data‑ingestion pipelines, limited source support, and high maintenance overhead, so it evaluated open‑source tools and adopted Apache SeaTunnel to unify batch and streaming data transfers, simplify ETL workflows, and provide a scalable, extensible solution for its multi‑TB daily data processing needs.
Shopee tackled frequent Flink checkpoint failures caused by back‑pressure by adopting and extending the community’s Unaligned Checkpoint mechanism—adding overdraft buffers, improving legacy sources, introducing an aligned‑checkpoint timeout, enabling output‑buffer switching, merging small HDFS files, and fixing network‑buffer deadlocks—now running hundreds of jobs with stable UC deployment and plans to enable it universally.
Shopee’s Merchant Services team created a visual, configurable onboarding platform that abstracts purchase flows into reusable templates and six modular configuration sections, moving product‑launch responsibilities from developers to operators, cutting code changes, boosting launch efficiency by about 75 % and adding import/export and validation tools.
Over three years, Shopee evolved its React Native architecture from a single‑bundle, centralized model to a fully decentralized system where each team independently develops, builds, and releases code, using an isolated JavaScript runtime and a self‑built Code Push Platform that manages permissions, incremental updates, grayscale releases, and migration of legacy plugins without disrupting ongoing business.
This article details Shopee Singapore Data Team’s implementation of a Flink‑based real‑time data warehouse, covering background challenges, layered architecture integrating Kafka, HBase, Druid, Hive, streaming pipelines, job management, monitoring, and future plans to expand Flink SQL support.