Designing Lampara: Ant Group’s Real‑Time Data Processing System for End‑to‑End SLA Guarantees

Business and Technical Background

Ant Group operates many real‑time scenarios—marketing (e.g., Five‑Blessing cards, flash‑sale, ad delivery), AI agents (e.g., Lingguang), and strong‑interaction services (live content, likes). These workloads face exploding data demand, increasing complexity, and strict stability and efficiency requirements, while the company previously lacked an effective end‑to‑end SLA solution.

Key Pain Points and Technical Challenges

Rapid growth of real‑time data demand and complexity (e.g., Five‑Blessing’s real‑time tags and features).

Real‑time data now drives critical decisions such as fund and traffic control.

Absence of an end‑to‑end SLA framework; existing industry solutions only cover component availability, not user‑visible latency.

Long, heterogeneous processing chains make SLA enforcement difficult.

Overall Strategy

In 2023 the data team launched a “from offline to real‑time” initiative, establishing a systematic, scenario‑driven development paradigm, a controlled‑processing platform, and a data‑research real‑time roadmap.

Lampara: Next‑Generation Real‑Time Controlled‑Processing System

Lampara is a fully managed, high‑availability, high‑performance data‑native processing system built on a pipeline model. Its core capabilities are:

End‑to‑end SLA guarantees, including dual‑link primary‑backup paths, full‑chain observability, and proactive disaster recovery.

Productized development paradigm that hides complex engineering choices from users.

Managed operations with automatic failover and DRE‑assisted manual intervention when needed.

Effective End‑to‑End SLA

During development, SLA is defined from the user perspective and decomposed into per‑engine targets. The system monitors latency at each stage, provides automatic failover (e.g., switching to a backup link when the primary flow fails), and records attribution for root‑cause analysis. In a recent Five‑Blessing campaign the active‑disaster‑recovery mechanism was triggered seven times without business impact.

Active Disaster Recovery

The platform distinguishes hot‑standby (both primary and backup run simultaneously) and cold‑standby (backup activated on failure via State‑based clustering). Hot‑standby runs on dedicated ECS resources, while cold‑standby can be launched quickly on a peer cluster.

Efficiency Under SLA Guarantees

Lampara’s scenario‑driven paradigm and enhanced operators (e.g., long‑period accumulation, AGG‑state‑compatible, materialized operators) lower the barrier for real‑time development and improve productivity. Users only specify high‑level business logic; the system auto‑generates tasks such as Flink one‑stage aggregation or two‑stage OLAP materialization.

Accuracy and Data Quality

Built‑in accuracy monitoring rules (primary‑backup checks, cross‑source validation, asset‑driven drift detection) are enforced as managed standards, reducing configuration errors and preventing risky operations. The system also provides a “god‑view” for DREs to detect downstream quality anomalies that indicate upstream issues.

Cost‑Effective High SLA

Compared with user‑built primary‑backup pipelines, Lampara reduces cost through shared hot‑standby resources and state‑based cold‑standby clusters. Resource isolation (e.g., exclusive ECS for primary, high‑availability nodes for backup) further lowers expense while meeting strict SLA demands.

Stage Progress

Business Applications

Lampara powers large‑scale scenarios such as flash‑sale, “碰一下”, Double‑Eleven, Five‑Blessing (3‑4× the previous scale), and AI agents, supporting Alipay, digital payment, AIRS, and insurance services.

Decision‑Making Efficiency

Real‑time dashboards can be delivered within 12 hours after asset creation, accelerating decision cycles from weeks to near‑real‑time.

Asset Reuse and Sustainability

The platform has built a middle‑layer covering 11 modules (traffic, placement, marketing, transaction, etc.), achieving high reuse rates and reducing duplicate development. Dynamic/static checks ensure long‑term maintainability of assets, automatically recommending controlled high‑SLA assets when user expectations cannot be met.

Stability Improvements

Lampara achieved a 99.5 % SLA fulfillment rate (100 % for managed tasks) and improved development efficiency by over 50 %.

Talent and Culture

Through battle‑training, mentorship, and multiple internal trainings, a cohort of real‑time data engineers has been cultivated, capable of independently handling real‑time projects.

Future Outlook

Ant Group plans to evolve toward AI‑native data infrastructure, exploring Fluss‑based lake‑stream integration, Paimon‑based multimodal storage, and exposing AI‑first interfaces (MCP, CLI, Skills). The goal is to make real‑time, big‑data, and AI capabilities as ubiquitous as water and electricity.