New Features in Flink 1.18: Operator-Level State TTL, Watermark Alignment, Idle Detection, and Dynamic Scaling

Flink 1.18 has been released, bringing many new features that are important for production environments.

Operator‑Level State TTL Settings

Starting with version 1.18, Table API and SQL users can set a Time‑to‑Live (TTL) for stateful operators individually using the configuration key table.exec.state.ttl. This allows different left‑ and right‑hand streams to have distinct TTLs, helping to control state size, reduce storage, and accelerate recovery after failures or restarts.

The official method involves editing a generated JSON plan file to specify the TTL for each operator. An example plan modification is shown below:

-- left source table
CREATE TABLE Orders (
    `order_id` INT,
    `line_order_id` INT
) WITH (
    'connector'='...'
);

-- right source table
CREATE TABLE LineOrders (
    `line_order_id` INT,
    `ship_mode` STRING
) WITH (
    'connector'='...'
);

-- sink table
CREATE TABLE OrdersShipInfo (
    `order_id` INT,
    `line_order_id` INT,
    `ship_mode` STRING
) WITH (
    'connector'='...'
);

COMPILE PLAN '/path/to/plan.json' FOR
INSERT INTO OrdersShipInfo
SELECT a.order_id, a.line_order_id, b.ship_mode 
FROM Orders a JOIN LineOrders b 
    ON a.line_order_id = b.line_order_id;

By modifying the JSON plan, users can assign different TTL values to the left and right streams, which is especially useful for large‑scale state management.

Watermark Alignment and Idle Detection

Previously, watermark alignment and idle‑time detection were only available through the DataStream API. In Flink 1.18 they can be configured via table options, for example:

-- configure in table options
CREATE TABLE user_actions (
  ...
  user_action_time TIMESTAMP(3),
  WATERMARK FOR user_action_time AS user_action_time - INTERVAL '5' SECOND
) WITH (
  'scan.watermark.idle-timeout'='1min',
  ...
);

-- use OPTIONS hint for alignment
SELECT ... FROM source_table
/** OPTIONS(
  'scan.watermark.alignment.group'='alignment-group-1',
  'scan.watermark.alignment.max-drift'='1min',
  'scan.watermark.alignment.update-interval'='1s'
) */

Watermark alignment ensures that watermarks generated by parallel source instances are synchronized, while idle detection allows different sources to have independent timeout settings, preventing downstream operators from stalling when some sources become silent.

Dynamic Fine‑Grained Scaling

From Flink 1.18 onward, the parallelism of any task can be changed at runtime through the Flink Web UI or REST API. This capability removes previous platform limitations, enabling easy scaling up or down of jobs, better resource utilization, and smoother integration with back‑pressure monitoring for stable cluster operation.

Additional improvements include better support for Paimon and other minor enhancements; users are encouraged to consult the official documentation for the full list of changes.