Real‑Time Member Level Calculation at Trillion‑Event Scale: A Production‑Ready Apache Flink Architecture

Why the System Is Hard

Member tiers drive discounts, points, coupons, dedicated support, and after‑sale priority in e‑commerce, retail, local services, and content platforms. Real‑time tier changes must appear within seconds, be precise (no duplicate upgrades or missed calculations), support frequent rule changes without job restarts, handle retroactive order corrections, and ingest up to hundreds of thousands of events per second, quickly growing state to terabytes.

Business Problem Abstraction

A mature tier system aggregates four metric groups: transaction (12‑month payment amount, order count, net refunds), activity (30‑day active days, consecutive sign‑ins, interaction counts), growth (effective invites, first‑purchase conversions, task completions), and risk (fraud tags, abnormal refund rate, blacklist status). Each metric has time windows, event‑type differences, correction rules, and triggers (upgrade/downgrade, benefit issuance, notifications, audit trails).

Why Batch Is Insufficient and Stream Is Suitable

Batch pipelines run hourly or daily, causing latency, high re‑processing cost for complex windows, and difficulty reconciling rule changes. Flink solves these issues with event‑time processing, keyed state per user, checkpoint‑based exactly‑once semantics, broadcast state for hot rule updates, and rich APIs (side outputs, async I/O, timers) for complex flows.

Production‑Level Architecture Overview

Design Principles

Event‑driven: all tier changes are triggered by events, not periodic batch overwrites.

State co‑location: user growth state is kept in Flink keyed by uid, avoiding external real‑time aggregation.

Rule decoupling: rules are separate, hot‑updatable decision tables.

Dual write output: one sink stores the current snapshot, another stores change trails for audit.

Re‑computable: historical corrections can be replayed without manual DB edits.

Observability: latency, backlog, checkpoint size, hot keys, and downgrade rates are all measurable.

Unified Event Model

public class MemberEvent { private String eventId; private String uid; private String eventType; private Long eventTime; private Long opTime; private String source; private BigDecimal payAmount; private BigDecimal refundAmount; private Integer inviteCount; private Integer activeScore; private String orderId; private String traceId; private Integer version; }

Key fields include eventId (global deduplication), eventTime (business time), opTime (ingestion time for latency analysis), and version (idempotent updates).

Member State Model

public class MemberLevelState { private String uid; private String currentLevel; private Integer currentLevelPriority; private BigDecimal rollingPayAmount12m; private BigDecimal rollingRefundAmount12m; private Integer rollingActiveDays30d; private Integer rollingInviteCount12m; private Long lastEvaluateTime; private Long lastLevelChangeTime; private String ruleVersion; private Boolean frozenByRisk; }

The state separates process data (rolling metrics), decision data (current level, freeze flag), and audit data (rule version, timestamps) to simplify explanation, back‑fill, and auditing.

Flink Job Topology

Source: consume order, payment, refund, task, rule, and replay topics.

Normalize: validate and standardize events, side‑output dirty data.

Dedup: drop duplicates using eventId.

Aggregate: keyed by uid to maintain long‑lived state.

Enrich: async join member profile, risk tags, organization info.

Evaluate: broadcast rule stream evaluates upgrades/downgrades.

Sink: write current snapshot, change events, and audit trails.

Why Not Use Sliding Windows for 12‑Month Metrics

Sliding a 365‑day window with 1‑day slide creates massive overlapping windows; state and compute explode at scale. The production approach splits the long window into incremental buckets (monthly for payment, daily for activity) and updates only the affected bucket per event.

Bucket State Design

public class MonthBucket { private Integer monthKey; private BigDecimal payAmount = BigDecimal.ZERO; private BigDecimal refundAmount = BigDecimal.ZERO; private Set<String> paidOrderIds = new HashSet<>(); }

public class DayBucket { private Integer dayKey; private Boolean active; }

This converts window overlap into incremental state updates, stabilizes memory growth, eases cleanup, fits RocksDB LSM storage, and avoids re‑building windows during replay.

Core Flink Mechanisms

Event Time

Tier calculations use eventTime (business time) rather than arrival time to handle delayed payments, third‑party sync lag, and retroactive corrections.

Watermark

Watermarks define when the system considers data before a timestamp complete. Typical tolerances: 30 s–3 min for payment streams, longer for back‑fill streams, with extremely late data routed to a side output for replay.

Keyed State

Per‑user state includes MapState<Integer, MonthBucket>, MapState<Integer, DayBucket>, ValueState<MemberLevelState>, and a version map to enforce idempotence.

Timer

Timers drive periodic bucket cleanup, rule re‑evaluation after rule effective times, and risk‑freeze expiration, ensuring state consistency even without new events.

Rule Hot‑Update Operator

public class LevelEvaluateProcessFunction extends KeyedBroadcastProcessFunction<String, MemberSnapshot, LevelRuleSet, LevelChangeEvent> { private static final MapStateDescriptor<String, LevelRuleSet> RULE_STATE = new MapStateDescriptor<>(