Design and Implementation of a Real-Time Marketing Automation Engine at vivo

This article is the fifth part of the series "vivo Marketing Automation Technology Revealed". It focuses on the background, value, architecture, and key technical designs of a real‑time marketing engine used in vivo's marketing automation business.

Background

Marketing automation can be divided into two main scenarios based on timeliness:

Offline batch targeting : Analyze offline data, generate user groups, and push messages such as new‑product recommendations, activity warm‑ups, periodic care, or user re‑engagement.

Real‑time personalized targeting : Track a single user's behavior within a short time window and send timely actions such as payment reminders or activity‑condition triggers.

Real‑time targeting requires higher scalability, low latency, and more complex data handling compared with offline batch processing.

Core Architecture Design

The system is divided into the following layers:

Access Layer : Provides multiple event ingestion methods (HTTP, Dubbo, MQ, etc.) and normalizes them to a unified MQ for downstream distribution.

Data Processing Layer : The real‑time engine core, responsible for data enrichment, metric calculation (often using Flink), user‑group matching, and rule matching.

Data Output Layer : Handles result distribution and delivery strategies.

Data Management : Stores event metadata configuration.

Data Warehouse : Persists offline data for further analysis.

Key Components and Design Practices

3.1 Event Access Scalability Design

Four queue design schemes are compared:

One queue per event (fine‑grained isolation, high maintenance cost).

One queue per source (moderate isolation, easier onboarding).

All sources share a single queue (low coupling, but poor isolation).

Priority‑based queues (high‑priority events get dedicated queues).

Conclusion: Scheme 2 (one queue per source) offers the best balance for the current project.

3.1.2 Dynamic Message Listening

Two optimization proposals are presented:

Load queue configuration from a file at startup – avoids redeployment but still requires a restart.

Store queue configuration in a database and use a scheduled task to listen for changes – eliminates both redeployment and restart.

Final decision: adopt the database‑driven dynamic listening (proposal 2).

3.2 Unified Distribution Processing

A generic dispatch template: event validation → EventMatcher → output channel. Reliability is ensured by configuring MQ to resend un‑acknowledged messages and making consumer logic idempotent.

3.3 Complex Multi‑Source Data Handling

Metric enrichment via a unified big‑data platform.

Metric calculation using Flink CEP for windowed processing.

Set operations (union, intersect, difference) performed with Presto.

3.4 Rule Engine

Traditional hard‑coded rule handling is contrasted with modern rule engines (BRMS). Advantages include separation of business rules from code, dynamic updates without redeployment, and better maintainability. Open‑source options such as Drools, Easy Rules, Aviator, Groovy, and Spring Expression Language are discussed, with guidance on when to choose a full‑featured BRMS (Drools) versus lightweight solutions (Easy Rules, Aviator).

Conclusion

The real‑time marketing engine architecture consists of event ingestion, data processing, metric computation, data output, metadata configuration, and data‑warehouse management. Queue isolation and dynamic configuration enable scalable event intake, unified dispatch abstracts processing flow, smooth start‑stop mechanisms guarantee reliability, and a rule engine improves configuration efficiency.