How RTS Platform Turns Real‑Time Data Streams into Reliable Business Value

Business Background

With the decline of internet growth, companies are shifting to refined, data‑driven operations. In advertising, the focus moves from merely winning impressions to optimizing pricing, making real‑time data crucial for decision‑making.

Technical Challenges of Real‑Time Computing

Stability equals timeliness : Real‑time engines must maintain sub‑hour latency and high throughput, as any instability directly impacts revenue.

Heterogeneous multi‑level computation : Unlike batch jobs, real‑time pipelines involve non‑homogeneous operator instances, strict exactly‑once semantics, and complex data dependencies that increase debugging difficulty.

Frequent schema changes : Data schemas act as interfaces between operators; constant additions of fields or types cause downstream breaks and large financial losses.

These issues make it hard for business teams to adopt real‑time solutions without incurring high operational overhead.

RTS Platform Overview

RTS (Real Time Stream) is a SaaS‑style platform that abstracts commercial real‑time data processing into scenario‑based development modes. Users express business logic with SQL and user‑defined functions (UDFs); the platform then automatically builds, tests, and deploys the optimal streaming topology.

Key capabilities include:

Auto schema change: All schema versions are stored in a code repository and released via CI/CD pipelines, ensuring ordered, coordinated updates.

Primary‑secondary redundant clusters: Critical data flows run on dual clusters with seamless failover and versioned rollbacks.

Experiment‑first workflow: Developers run experiments on a per‑field basis, then push successful changes to production with a single click.

Metadata generation: Integrated with LinkedIn DataHub to capture data lineage, enabling impact analysis and automated testing.

Scenario‑Based Development

Real‑time decision making requires fast schema evolution, frequent new fields, and custom UDFs. RTS abstracts this as an “import‑export” model where users register data sources, define tables, write UDFs, and create import/export tasks using SQL.

Conversion business (tracking ad conversions) demands deduplication, tagging, and high availability. RTS deploys the tagging and deduplication modules on redundant clusters and uses a switcher component to ensure idempotent processing across primary and backup streams.

Operational Flow

Create data source metadata and verify permissions.

Define tables and set garbage‑collection policies.

Develop and register custom UDFs.

Build import tasks (ETL) and export tasks (to HDFS, message queues, etc.).

Add new fields by updating schema; RTS automatically generates dependent jobs.

All steps are managed through the platform’s UI, shielding users from underlying engine complexities.

Current Impact and Future Outlook

RTS now supports conversion, real‑time decision, real‑time data‑warehouse dashboards, data distribution, word‑list delivery, and data mining scenarios, delivering millions of dollars in daily revenue. Future plans include expanding to full‑stream data stitching for all commercial flows and integrating generative AI models to automate data‑driven insights.

Illustrations