Real-time Log Aggregation and Monitoring with Blink (Flink) on Mobile Endpoints

This article introduces Blink, an Alibaba‑derived variant of Apache Flink, and describes how it is used for real‑time aggregation and cleaning of mobile endpoint telemetry.

Flink originated as a research project at Berlin Institute of Technology, evolved from batch processing to a unified stream‑and‑batch engine. Blink inherits Flink’s runtime and adds optimizations for Alibaba’s large‑scale use cases.

Key concepts include dynamic tables, which treat unbounded streams as continuously changing tables, enabling SQL‑based processing of both bounded and unbounded data.

Example DDL for a source table used by the UserTrack (UT) system:

CREATE TABLE s_ut (
  filed1 VARCHAR COMMENT '字段1注释',
  filed2 VARCHAR COMMENT '字段2注释',
  ...
) WITH (
  type = 'ut',
  topic = 's_ut',
  filterList = 'app_bu=''TMYP''',
  nullValues = '\\N|',
  maxFetchSize = '100'
);

Corresponding sink table definition for Alibaba Cloud Log Service (SLS):

CREATE TABLE sls_tmyp (
  filed1 VARCHAR COMMENT '字段1注释',
  filed2 VARCHAR COMMENT '字段2注释',
  ...
) WITH (
  type = 'sls',
  endPoint = '阿里云SLS服务地址',
  project = 'SLS空间名',
  logStore = 'SLS日志库名'
);

A view can be created to compute per‑minute API success rates using a tumbling window:

CREATE VIEW v_yp_api_rate AS
SELECT
  TUMBLE_END(time_field, INTERVAL '1' MINUTE) AS `time`,
  -- additional aggregations such as SUM(success) / SUM(total) ...
FROM s_ut
WHERE os = 'android';

The result is written to the sink with an INSERT statement:

INSERT INTO sls_tmyp
SELECT field1, field2, ...
FROM v_yp_api_rate;

By chaining these steps, the UT pipeline transforms raw mobile telemetry into a structured real‑time monitoring dashboard, supporting use cases such as API success rates, page exposure, performance metrics, and more.

The same streaming‑SQL approach can be extended to fraud detection, IoT device management, and other domains.