Understanding Continuous Queries in Apache Flink: From Static Queries to Dynamic Tables and Trigger Simulations

In stream processing, data arrives continuously and never ends, so the computation also never terminates. The article introduces how Apache Flink uses continuous queries to continuously output results from an unending data flow.

Data Management – Before discussing continuous queries, the article compares Flink’s data abstraction (source and sink) with traditional databases such as MySQL, highlighting that Flink separates storage and computation while still supporting ANSI‑SQL.

Static Query – A traditional database query (e.g., select * from flink_tab;) returns a final result once and then stops. The result is static because the table content does not change during the query.

Continuous Query – In Flink, a continuous query runs on a dynamic table that keeps changing; each new record triggers an incremental recomputation, producing an ever‑updating result.

Static vs. Continuous Characteristics – A table summarizing the differences shows that static queries compute once, while continuous queries compute infinitely and continuously update the result.

Relationship – Each execution of a continuous query can be seen as a static query at a specific point in time. Flink implements this by incrementally applying the new record to the previous result.

Simulating Continuous Queries with MySQL Triggers

Using MySQL triggers, the article demonstrates two scenarios:

Append‑only (no primary key) – After each INSERT, a trigger writes the current table snapshot to a file, producing an ever‑growing result set.

Update (with primary key) – After INSERT or UPDATE, a trigger writes the snapshot, showing how results are updated rather than only appended.

SQL examples are shown in pre blocks, preserving the original statements.

How Flink Achieves Continuous Queries

Flink treats each incoming event as a new computation step: result(n) = calculation(result(n‑1), n). Incremental aggregation (e.g., count(id), sum(amount)) stores intermediate state and updates it with each event.

For update‑heavy tables, Flink uses a retraction mechanism: events are marked as insert or delete, and aggregate functions provide accumulate and retract methods to adjust the stored counts correctly.

Connector Types

Flink abstracts external storage via connectors, divided into Source (read) and Sink (write). The article discusses the lack of primary‑key support in source connectors and the impact on join operations, where duplicate keys cause unnecessary matching and resource pressure.

Solution for PK‑Based Dynamic Tables – By applying a GROUP BY with LAST_VALUE (an Alibaba‑specific extension) after the source, a dynamic table with a primary key can be created, enabling efficient upserts and reducing computation load.

Sink Modes

Append mode – only INSERT statements are generated when no primary key is defined.

Upsert mode – INSERT/UPDATE/DELETE statements are generated when a primary key exists; UPDATE is performed per key.

Retract mode – both INSERT and DELETE messages are emitted, and the sink connector translates them into appropriate storage operations.

Conclusion – The article uses MySQL triggers to illustrate static vs. continuous query concepts, explains Flink’s incremental processing and retraction mechanisms, and shows how proper primary‑key handling and connector configuration can optimize continuous query workloads in big‑data stream applications.