This article explains how to develop a real‑time ETL application using Apache Flink that reads events from Kafka, partitions them by event time into HDFS directories, and achieves exactly‑once processing through checkpointing, custom bucket assigners, and proper state backend configuration.
The three‑day Flink Forward Asia 2019 conference in Beijing attracted over 2,000 attendees, showcased more than 45 talks from leading companies and researchers, and highlighted the evolution of Flink toward a unified engine, Stateful Functions, AI integration, cloud‑native deployment, and real‑time analytics at massive scale.
This article demonstrates how to develop a real‑time ETL job using Apache Flink, covering project setup, Kafka as a source, custom bucket assigners for HDFS, checkpointing, savepoints, and deployment on YARN to achieve exactly‑once processing guarantees.
This guide provides a detailed overview of Apache Flink covering its core streaming engine, APIs (DataSet, DataStream, Table), architectural components, comparison with Spark Streaming, partitioning, parallelism, restart strategies, state backends, time semantics, watermarks, SQL processing, fault‑tolerance mechanisms, memory management, serialization, RPC framework, back‑pressure handling, operator chaining, and practical tips for interview preparation.
This article presents a comprehensive collection of Flink interview questions covering fundamental concepts, advanced topics, and source‑code details to help candidates prepare effectively for Flink‑related technical interviews.
This article compiles a curated list of the most reliable and official Apache Flink learning materials—including beginner tutorials, source‑code walkthroughs, advanced topics, community articles, real‑world case studies, and downloadable resources—providing a one‑stop reference for developers and researchers interested in stream processing and big‑data analytics.
Apache Flink 1.9.0, released on August 22, merges Alibaba's Blink engine, introduces a major architecture overhaul, enriches Table API & SQL, adds batch and stream processing enhancements, and integrates tightly with Hive, marking a significant milestone for large‑scale data processing.
This article provides an in‑depth analysis of Stream SQL theory, incremental query algorithms, materialized view maintenance, optimizer cost models, time handling, windowing, and the practical capabilities and limitations of Apache Flink’s streaming SQL engine.
This article explains how to register scalar UDFs, table‑valued UDTFs, and aggregate UDAFs in Apache Flink, illustrates typical compilation and runtime pitfalls with concrete Scala code examples, and provides corrected implementations and best‑practice tips for reliable function registration.
This article explains the concept of Watermarks in stream processing, detailing their background, theoretical foundations from the Dataflow model, practical implementation in Apache Flink with code examples, and discusses trade‑offs between latency and accuracy for real‑time analytics.
This article explains how Apache Pulsar and Apache Flink can be combined to provide a unified, scalable, and fault‑tolerant data processing platform, covering Pulsar's architecture, its differences from other messaging systems, various integration patterns, and concrete code examples for stream and batch workloads.
On June 29, the Apache Flink Beijing Meetup presented a comprehensive analysis of Flink 1.9’s major architectural changes, new Table API & SQL capabilities, runtime and core enhancements, and future roadmap, with slides and resources made available for download.
This article provides an in‑depth overview of Flink SQL, covering its origins, the latest 1.7.0 and 1.8.0 enhancements, the underlying programming model, common operators and built‑in functions, and a complete end‑to‑end example that analyzes NBA scoring‑leader data using Flink SQL.
This article explains how Apache Flink's State TTL feature works, demonstrates configuring TTL for state size control and automatic cleanup, and details the continuous cleanup mechanisms introduced in Flink 1.8.0 for both heap and RocksDB state backends.
Apache Flink 1.8.0 introduces incremental state cleanup with TTL, updates Hadoop support, deprecates TableEnvironment static methods, adds new Kafka deserialization schema, modifies Maven dependencies, and provides several configuration and Table API enhancements for better stream‑processing performance and compatibility.
This article explains how Apache Flink’s TwoPhaseCommitSinkFunction, introduced in version 1.4, enables end-to-end exactly-once semantics when integrated with Apache Kafka, detailing the checkpoint mechanism and the two-phase commit protocol that ensures reliable data processing.
This article presents Gong Yunfei's 2019 talk from the Beijing Apache Flink x Apache Kafka conference, detailing how ByteDance leverages Kafka for its large‑scale streaming and data processing needs.
By iteratively redesigning the fatigue‑event detection pipeline and leveraging Apache Flink’s stateful stream processing, the team reduced network overhead, cut resource usage to a third, and achieved a stable 45,000 TPS throughput on six containers with 20 GB memory, while outlining three optimization phases and practical lessons.
This article compares Apache Spark and Apache Flink, outlining their programming models, streaming mechanisms, state management, time semantics, and exactly‑once guarantees, and highlights the strengths and differences of each framework for batch and real‑time big‑data processing.
This week's digest shares a personal anecdote and a series of technical deep‑dives into Apache Flink, covering JOIN LATERAL, TimeInterval JOIN, Temporal Table JOIN, state management, and related code examples, while also previewing upcoming work schedules and recommended Flink reference articles.
Pravega, Dell’s native stream storage project, addresses the challenges of modern low‑latency, exactly‑once stream processing by combining tiered storage, Apache BookKeeper, and seamless Flink integration, offering a unified solution that reduces development, storage, and operational costs compared to traditional message systems like Kafka.
This article explains how Apache Flink uses state for incremental stream processing, describes the different state backends, details the persistence mechanism, and shows how both OperatorState and KeyedState are redistributed during scaling using partition and key‑group algorithms.
This article explains the concept of Temporal Table JOIN, its implementation in SQL Server and Apache Flink, provides DDL/DML examples, compares it with other join types, and discusses improvements to align Flink with ANSI‑SQL standards.
This article explains how Apache Flink's Interval Join solves time‑bounded join requirements more efficiently than unbounded joins, covering its syntax, semantics, state‑management considerations, and providing a complete Scala example with code and execution results.
This article explains the special JOIN LATERAL operator, compares it with INNER JOIN and correlated subqueries, shows how SQL Server implements it via CROSS APPLY, and demonstrates its support in Apache Flink using Calcite and user‑defined table functions with concrete code examples.
This weekly briefing introduces Apache Flink's continuous query mechanism, demonstrates how to integrate Kafka as a DataStream connector, provides an overview of Flink SQL features, explains the implementation and optimization of dual‑stream JOIN operators, and showcases the Table API with end‑to‑end examples.
This article explains the purpose and types of SQL JOIN operators, demonstrates their syntax and semantics with examples, compares traditional database joins to Apache Flink's streaming two‑stream join implementation, and discusses optimization techniques such as state management, shuffle handling, and join reordering.
This article provides an in-depth introduction to SQL, its history and ANSI standards, then details Apache Flink's SQL capabilities—including SELECT, WHERE, GROUP BY, UNION, JOIN, window functions, and user-defined functions—accompanied by extensive code examples and a complete end‑to‑end Flink job implementation.
This article provides a step‑by‑step guide on installing Apache Kafka, creating topics, producing and consuming messages via command line, and demonstrates how to integrate Kafka with Apache Flink using the Flink‑Kafka connector, custom serialization schemas, and event‑time window processing.
This article explains how Apache Flink implements continuous queries for unbounded stream processing, compares static and continuous query semantics, demonstrates how MySQL triggers can simulate continuous queries in append‑only and update scenarios, and discusses Flink's connector, source, sink, and retraction mechanisms for correct incremental computation.
This article explains Apache Flink's fault‑tolerance mechanisms, including checkpointing, barrier alignment, the differences between At‑Least‑Once and Exactly‑Once semantics, configuration options, incremental checkpointing, and the requirements for external sources and sinks to achieve end‑to‑end exactly‑once processing.
This article explains Apache Flink’s fundamental design principle that treats batch as a special case of stream, compares native streaming with micro‑batching, describes its deployment modes, fault‑tolerance mechanisms, unified data and scheduling layers, and outlines Alibaba’s architectural optimizations for the platform.
This article explains how Apache Flink uses Watermarks to manage event‑time windows, describes the three time semantics, details periodic and punctuated Watermark generation methods with their Java interfaces, and shows practical DDL examples for handling late and out‑of‑order data in stream processing.
This tutorial explains how to deploy Apache Flink on a Hadoop YARN cluster, covering both YARN session mode and direct job submission, and demonstrates running the built‑in WordCount example with command‑line options for input, output, and resource configuration.
The 2018 Flink Forward China conference in Beijing showcased Apache Flink’s evolution, Alibaba’s massive contributions—including the Blink fork, real‑time BI, online learning and city‑level analytics—and highlighted how industry leaders like Alibaba, Didi and others leverage Flink for scalable, low‑latency big‑data processing across diverse use cases.
This article introduces Apache Flink’s rapid rise as the leading open‑source big data engine, explains its role in batch, stream, and interactive analytics, showcases real‑world use cases from Alibaba, Didi, and ByteDance, and outlines how Flink powers both big data and AI workloads.
This article explains how Alibaba built a comprehensive Blink testing platform—including code‑quality checks, functional, performance, stability, and pre‑release testing—to ensure the reliability and scalability of its real‑time big‑data processing engine during massive workloads like Double 11.
This article explains how Alibaba adopted Apache Flink as a unified, low‑latency, high‑throughput big‑data engine, detailing its stream‑first design, state management, checkpointing, massive production deployment, community contributions, and upcoming plans for a unified API, SQL layer, broader language support, and AI integration.
This article explains how Alibaba adopted Apache Flink as a unified, low‑latency, high‑throughput big data engine, covering its origins, technical advantages over Spark, large‑scale deployment, state management, checkpointing, API unification, and future directions in streaming and batch processing.
This article explains how the data aggregation platform adopts Apache Flink for high‑throughput, low‑latency stream processing, covering the complete workflow from data source integration, transformation operations, windowing and time concepts, to a concrete order‑count example with custom aggregation logic.
The study benchmarks Apache Flink against Apache Storm on a shared cluster, showing Flink delivering three‑to‑five times higher throughput and roughly half the latency across simple, sleep‑induced, and windowed workloads, with modest throughput loss for exactly‑once semantics, leading to a recommendation of Flink for high‑performance, stateful real‑time stream processing.
This article introduces Flink’s background, architecture, and ecosystem, then demonstrates a step‑by‑step tutorial on using Flink SQL to clean and transform streaming data from Kafka, highlighting its advantages over Storm for real‑time ETL.
The article recounts the 2017 Flink Forward conference in San Francisco, highlighting key sessions from DataArtisans, Uber, Netflix and Alibaba, and discusses real‑time stream processing use cases, large‑scale deployments, runtime and TableAPI/SQL improvements, and the growing adoption of Flink in the industry.
This article examines Apache Flink's stream‑processing architecture, compares its native streaming model, fault‑tolerance, performance and SQL capabilities with Spark and Storm, and concludes that Flink offers a more powerful and efficient solution despite some maturity gaps.
