Master Flink Optimizations: TTL, Mini‑Batch, Two‑Phase Aggregation, Lookup Join & More

This article reviews the most effective Flink optimization techniques since 2022, including operator‑level TTL, mini‑batch processing, two‑phase aggregation, multi‑dimensional DISTINCT with FILTER, lookup join caching strategies, and TopN implementations, each rated with recommendation stars for production use.

Big DataFlinkLookup Join

0 likes · 8 min read

Master Flink Optimizations: TTL, Mini‑Batch, Two‑Phase Aggregation, Lookup Join & More

Big Data Technology & Architecture

Feb 23, 2022 · Big Data

Understanding Mini‑Batch Streaming Aggregation in Flink SQL

This article explains Flink SQL’s streaming aggregation Mini‑Batch feature, covering its purpose, configuration parameters, underlying optimizer rules, operator implementations, watermark handling, buffer processing, and the optional Local‑Global two‑phase aggregation optimization for improving throughput and reducing state overhead in large‑scale data pipelines.

Big DataFlinkMini-Batch

0 likes · 10 min read

Understanding Mini‑Batch Streaming Aggregation in Flink SQL

Big Data Technology & Architecture

Nov 9, 2019 · Big Data

Comparative Study of Apache Flink and Spark Streaming at Xiaomi: Architecture, Performance, and Serialization

This article examines Xiaomi's migration from Spark Streaming to Apache Flink, comparing scheduling strategies, mini‑batch versus true streaming, resource utilization, latency, and serialization mechanisms, and concludes with practical insights and custom optimization techniques for large‑scale data processing.

Big DataFlinkMini-Batch

0 likes · 17 min read

Comparative Study of Apache Flink and Spark Streaming at Xiaomi: Architecture, Performance, and Serialization

360 Tech Engineering

Sep 16, 2019 · Artificial Intelligence

Backpropagation Algorithm for Fully Connected Neural Networks with Python Implementation

This article explains the backpropagation training algorithm for fully connected artificial neural networks, detailing its gradient‑descent basis, mathematical derivation, matrix formulation, and provides a complete Python implementation with mini‑batch stochastic gradient descent, momentum, learning‑rate decay, and experimental results.

BackpropagationMini-BatchNeural Network

0 likes · 14 min read

Backpropagation Algorithm for Fully Connected Neural Networks with Python Implementation

Hulu Beijing

Jan 30, 2018 · Artificial Intelligence

Understanding Stochastic Gradient Descent and Mini‑Batch Optimization

This article explains why traditional gradient descent struggles with massive datasets, introduces stochastic gradient descent and mini‑batch gradient descent as efficient alternatives, and provides practical guidance on batch size selection, data shuffling, and learning‑rate scheduling for deep learning models.

Mini-Batchoptimizationstochastic gradient descent

0 likes · 8 min read

Understanding Stochastic Gradient Descent and Mini‑Batch Optimization