Tagged articles

Flink

1000 articles · Page 9 of 10
Beike Product & Technology
Beike Product & Technology
Jun 12, 2020 · Big Data

Design and Implementation of SQL on Streaming (SQL 1.0 → SQL 2.0) in a Real‑Time Computing Platform

This article describes the evolution of a real‑time computing platform from SQL 1.0 built on Spark Structured Streaming to SQL 2.0 powered by Flink‑SQL, covering dynamic tables, continuous queries, dimension‑table joins, cache optimization, DDL extensions, platformization, operational challenges and future roadmap.

Dimension TableFlinkReal-Time Computing
0 likes · 19 min read
Design and Implementation of SQL on Streaming (SQL 1.0 → SQL 2.0) in a Real‑Time Computing Platform
DataFunTalk
DataFunTalk
Jun 11, 2020 · Big Data

Real-time Multi-dimensional Analytics and SlimBase State Backend at Kuaishou: Flink Applications and Optimizations

This article presents Kuaishou's extensive use of Apache Flink for real-time multi-dimensional analytics, detailing the platform's architecture, cluster scale, data processing pipelines, the design of a shared state storage engine called SlimBase, and performance improvements achieved through replacing RocksDB with a customized HBase‑based solution.

FlinkKuaishouReal-time Analytics
0 likes · 15 min read
Real-time Multi-dimensional Analytics and SlimBase State Backend at Kuaishou: Flink Applications and Optimizations
Architect
Architect
Jun 10, 2020 · Big Data

Understanding Flink Time Notions: ProcessTime, EventTime, IngestionTime and Watermarks with Code Examples

This article explains the three time notions supported by Apache Flink—ProcessTime, EventTime, and IngestionTime—detailing their semantics, how Watermarks enable event‑time processing, and provides Scala code samples for configuring time characteristics, assigning timestamps, and generating Watermarks in a streaming job.

EventTimeFlinkScala
0 likes · 16 min read
Understanding Flink Time Notions: ProcessTime, EventTime, IngestionTime and Watermarks with Code Examples
58 Tech
58 Tech
Jun 10, 2020 · Big Data

Real‑time Data Warehouse Practices at 58 Tongcheng Bao: From Spark Streaming 1.0 to Flink‑based 2.0

This article details the evolution of 58 Tongcheng Bao's real‑time data warehouse, describing the initial Spark‑Streaming architecture, its limitations, and the redesign using Flink with a layered ODS‑DWD‑DWS‑APP model, data‑quality monitoring, join techniques, and the resulting improvements in latency and accuracy.

Data QualityFlinkKafka
0 likes · 9 min read
Real‑time Data Warehouse Practices at 58 Tongcheng Bao: From Spark Streaming 1.0 to Flink‑based 2.0
dbaplus Community
dbaplus Community
Jun 2, 2020 · Big Data

How Cainiao Built a Scalable Real‑Time Data Warehouse with Flink

Facing growing order volumes and strict timeliness demands, Cainiao’s tech team overhauled its real‑time data warehouse by redesigning data models, adopting Flink for streaming computation, upgrading data services, and exploring innovative tools, sharing practical lessons and future directions for large‑scale logistics analytics.

FlinkStreamingbig data
0 likes · 18 min read
How Cainiao Built a Scalable Real‑Time Data Warehouse with Flink
Architect
Architect
May 30, 2020 · Big Data

Understanding Flink’s Unified Programming API for Batch and Streaming Jobs

This article examines Apache Flink’s programming model, comparing its batch DataSet API with the streaming DataStream API, detailing class hierarchies, key code examples such as groupBy and job submission, and explaining how both paradigms are unified into a common JobGraph representation.

Batch ProcessingFlinkJobGraph
0 likes · 9 min read
Understanding Flink’s Unified Programming API for Batch and Streaming Jobs
Architect
Architect
May 29, 2020 · Artificial Intelligence

Integrating Flink with TensorFlow for End-to-End Machine Learning Pipelines

This article explains how to combine the Flink data‑processing engine with TensorFlow to create a unified, end‑to‑end machine‑learning workflow, covering background, challenges, the Flink‑AI‑extended architecture, ML framework and operator abstractions, and both batch and streaming training and prediction modes.

AI IntegrationDistributed TrainingFlink
0 likes · 9 min read
Integrating Flink with TensorFlow for End-to-End Machine Learning Pipelines
Huolala Tech
Huolala Tech
May 28, 2020 · Big Data

How Flink Powers Real‑Time Risk Control at HuoLaLa: Architecture and Insights

This article explains Flink's role in HuoLaLa's risk‑control system, covering its background, the Lambda‑style architecture that combines batch and streaming, the real‑time data pipeline, machine‑learning models, and operational safeguards that together enable proactive fraud detection.

Big Data ArchitectureFlinkLambda Architecture
0 likes · 16 min read
How Flink Powers Real‑Time Risk Control at HuoLaLa: Architecture and Insights
DataFunTalk
DataFunTalk
May 14, 2020 · Big Data

Building a Real-Time Data Warehouse at Cainiao: Architecture, Model Upgrades, Engine Enhancements, and Service Innovations

This article shares Cainiao's practical experience in constructing a real-time data warehouse, covering the shortcomings of the previous architecture, the evolution of data models, the migration to Flink with advanced features like retraction and timer services, and the modernization of data services and tooling to support high‑throughput logistics scenarios.

Data ServiceFlinkReal-time Data Warehouse
0 likes · 16 min read
Building a Real-Time Data Warehouse at Cainiao: Architecture, Model Upgrades, Engine Enhancements, and Service Innovations
Big Data Technology Architecture
Big Data Technology Architecture
Apr 15, 2020 · Big Data

Real-Time Data Warehouse Practices: Case Studies from Meituan, NetEase, Zhihu, and OPPO

This article reviews the evolution of data warehouses from traditional offline models to modern real‑time architectures, presenting detailed case studies of Meituan, NetEase, Zhihu, and OPPO, and discusses layer designs, technology choices such as Flink, Kafka, and storage options, and key lessons for building scalable real‑time warehouses.

FlinkKafkabig data
0 likes · 13 min read
Real-Time Data Warehouse Practices: Case Studies from Meituan, NetEase, Zhihu, and OPPO
Dada Group Technology
Dada Group Technology
Apr 15, 2020 · Big Data

Practice Experience of Dada Group's Real-Time Computation SQLization Using Dada Flink SQL

This article details Dada Group's development of the Dada Flink SQL engine, describing its background, architecture, parser design, dimension‑table join strategies, numerous enhancements such as HA support, Kafka keyword handling, metadata integration, Redis and ClickHouse sinks, BINLOG simplification, and future migration plans toward Flink 1.10.

ClickHouseFlinkReal-Time Computing
0 likes · 12 min read
Practice Experience of Dada Group's Real-Time Computation SQLization Using Dada Flink SQL
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 8, 2020 · Big Data

Common Apache Flink Exceptions and How to Resolve Them

This article enumerates typical Apache Flink deployment, job, and checkpoint errors—such as JDK version issues, resource shortages, task manager timeouts, and state migration problems—and provides practical troubleshooting steps and configuration tips to help engineers quickly diagnose and fix these failures.

CheckpointExceptionFlink
0 likes · 8 min read
Common Apache Flink Exceptions and How to Resolve Them
DataFunTalk
DataFunTalk
Mar 28, 2020 · Big Data

Applying Flink State Management for Real-Time Recommendation Scenarios

This article explains how Apache Flink's flexible state management can be leveraged to solve data correlation challenges in real‑time recommendation platforms, compares Flink with Spark and Storm, describes the underlying broadcast and managed state mechanisms, and provides a step‑by‑step implementation using Kafka, Druid, and custom broadcast functions.

FlinkStreamingbig data
0 likes · 14 min read
Applying Flink State Management for Real-Time Recommendation Scenarios
Alibaba Cloud Developer
Alibaba Cloud Developer
Mar 19, 2020 · Big Data

Can Flink Unify Real‑Time and Offline Data Warehouses? A Deep Dive

This article examines the challenges of maintaining separate offline and real‑time data warehouses, explains the three‑layer ODS‑DW‑ADS model, evaluates the traditional Lambda architecture, and explores how a unified Flink stack with Kafka, HiveCatalog and streaming sinks can simplify metadata, SQL development, data import/export, and stateful processing for both batch and streaming workloads.

FlinkLambda Architecturedata warehouse
0 likes · 12 min read
Can Flink Unify Real‑Time and Offline Data Warehouses? A Deep Dive
DataFunTalk
DataFunTalk
Mar 8, 2020 · Big Data

Real-Time Log Monitoring and Alerting System for iQIYI Membership Services

This article describes how iQIYI built a real‑time, multi‑dimensional log monitoring platform using Spark Streaming, Flink, Kafka and Druid to handle billions of logs, improve alerting accuracy, reduce incident response time, and outline future intelligent monitoring enhancements.

DruidFlinkLog Analytics
0 likes · 10 min read
Real-Time Log Monitoring and Alerting System for iQIYI Membership Services
iQIYI Technical Product Team
iQIYI Technical Product Team
Mar 6, 2020 · Big Data

Real-Time Log Monitoring and Alerting for iQIYI Membership Services

To support over 100 million iQIYI members, the team rebuilt a real‑time log monitoring platform that gathers access, exception, Nginx and front‑end logs via a Venus‑Agent, streams them through Kafka to Spark Streaming and Flink, stores metrics in Druid, and provides minute‑level host and business alerts, achieving 80 % faster incident investigation, detecting 90 % of member complaints early, and generating more than 4,800 actionable alerts.

FlinkLog AnalyticsSpark Streaming
0 likes · 11 min read
Real-Time Log Monitoring and Alerting for iQIYI Membership Services
58 Tech
58 Tech
Mar 4, 2020 · Big Data

Applying Flink State Management to Real‑Time Recommendation Scenarios

This article explains how Flink's flexible state management, including Broadcast, Keyed, and Operator states, can be used to solve real‑time recommendation challenges such as per‑minute UV, click, and exposure counting, while addressing locality mapping and data‑delay issues with Druid as the downstream store.

Broadcast StateDruidFlink
0 likes · 13 min read
Applying Flink State Management to Real‑Time Recommendation Scenarios
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 22, 2020 · Big Data

Understanding Flink's Asynchronous Barrier Snapshot (ABS) Algorithm for Checkpointing

This article explains how Apache Flink implements fault‑tolerant checkpointing using the Asynchronous Barrier Snapshot (ABS) algorithm, a localized version of the Chandy‑Lamport distributed snapshot, covering barriers, snapshot alignment, exactly‑once versus at‑least‑once semantics, and handling of cyclic dataflow graphs.

Asynchronous Barrier SnapshotCheckpointingFlink
0 likes · 9 min read
Understanding Flink's Asynchronous Barrier Snapshot (ABS) Algorithm for Checkpointing
Alibaba Cloud Developer
Alibaba Cloud Developer
Feb 19, 2020 · Artificial Intelligence

How Flink Is Powering Real‑Time AI: From Lambda Architecture to Stream‑Batch Unification

This article examines how Apache Flink embraces AI by leveraging the Lambda architecture and stream‑batch unification to enable real‑time data processing across preprocessing, model training, and inference, discusses the challenges of model updates and code maintenance, and outlines ongoing Flink initiatives that support AI real‑timeization.

AIFlink
0 likes · 15 min read
How Flink Is Powering Real‑Time AI: From Lambda Architecture to Stream‑Batch Unification
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 15, 2020 · Big Data

Understanding Event Time and Watermarks in Apache Flink

This article explains how Apache Flink uses event‑time timestamps and watermarks to handle out‑of‑order and late data, describes the assignTimestampsAndWatermarks API with periodic and punctuated watermark assigners, and provides practical code examples for window lateness and side‑output handling.

Apache FlinkEvent TimeFlink
0 likes · 10 min read
Understanding Event Time and Watermarks in Apache Flink
Big Data Technology Architecture
Big Data Technology Architecture
Feb 13, 2020 · Big Data

Evolution of Cainiao's Real-Time Data Warehouse Architecture: Model, Compute Engine, and Data Service Upgrades

The talk details Cainiao’s evolution of its real‑time data warehouse architecture, covering the original 2016 model, compute and service challenges, the 2017 multi‑layer data model redesign, migration to Flink, practical cases of state retraction, timeout statistics, smart optimizations, and the unified data service platform.

Data ServiceFlinkReal-time Data Warehouse
0 likes · 16 min read
Evolution of Cainiao's Real-Time Data Warehouse Architecture: Model, Compute Engine, and Data Service Upgrades
Xianyu Technology
Xianyu Technology
Feb 11, 2020 · Big Data

Client-side Complex Event Processing with Flink CEP and Python

The article describes how Xianyu’s recommendation system shifts complex event processing from server‑side Blink to client‑side Python using Flink CEP concepts, detailing the NFA‑based state and transition model, pattern‑building API, aggregation support, achieving sub‑second execution with modest memory, and outlines future optimizations such as NFA persistence, windowing, DSL script generation, and C++/TensorFlow Lite acceleration.

CEPClientSideFlink
0 likes · 13 min read
Client-side Complex Event Processing with Flink CEP and Python
DataFunTalk
DataFunTalk
Feb 10, 2020 · Artificial Intelligence

Real‑Time Intelligent Anomaly Detection Platform at Ctrip: Integrating Flink and TensorFlow (Prophet)

The article describes Ctrip's Prophet platform, which combines Flink real‑time stream processing with TensorFlow deep‑learning models to provide intelligent, low‑latency anomaly detection, replacing traditional rule‑based alerts and addressing challenges such as holiday traffic and model scalability.

AIFlinkTensorFlow
0 likes · 13 min read
Real‑Time Intelligent Anomaly Detection Platform at Ctrip: Integrating Flink and TensorFlow (Prophet)
Big Data Technology Architecture
Big Data Technology Architecture
Feb 8, 2020 · Big Data

Meituan-Dianping Real-Time Data Warehouse Platform Built on Apache Flink: Architecture, Practices, and Future Directions

Meituan-Dianping’s senior technical expert shares the evolution, architecture, and implementation of their Apache Flink‑based real‑time data warehouse platform, covering platform evolution, layered design, job and resource management, business warehouse use cases, and future development considerations.

FlinkMeituan-DianpingReal-time Data Warehouse
0 likes · 16 min read
Meituan-Dianping Real-Time Data Warehouse Platform Built on Apache Flink: Architecture, Practices, and Future Directions
DataFunTalk
DataFunTalk
Jan 22, 2020 · Big Data

Real-Time Data Engineering Practices for Alibaba 1688 Business

This article explains how Alibaba 1688 achieves real‑time recommendation, advertising, and product statistics through a robust middle‑platform foundation, streaming engines like Blink, data synchronization tools, and scalable storage, illustrating three concrete engineering cases and the end‑to‑end real‑time data service pipeline.

AlibabaFlinkstream processing
0 likes · 8 min read
Real-Time Data Engineering Practices for Alibaba 1688 Business
Alibaba Cloud Developer
Alibaba Cloud Developer
Jan 20, 2020 · Big Data

Alibaba’s Secrets to High‑Throughput Full‑Load and Low‑Latency Search Processing

This article details how Alibaba migrated its massive Taobao‑Tmall search workload to the search offline platform, tackling challenges of massive data volume, one‑to‑many joins, and hotspot sellers through a series of performance optimizations—including local joins, salt‑based data sharding, dynamic aggregation jobs, and asynchronous processing—to achieve high‑throughput full loads and low‑latency incremental updates.

AlibabaFlinkbig data
0 likes · 15 min read
Alibaba’s Secrets to High‑Throughput Full‑Load and Low‑Latency Search Processing
dbaplus Community
dbaplus Community
Jan 14, 2020 · Big Data

How OPPO Built a Real‑Time Data Warehouse with Flink SQL

This article details{32-64 words} OPPO's evolution from an offline data warehouse to a real‑time platform, describing the business scale, data‑mid platform architecture, migration strategy using Flink SQL, extensions like AthenaX, and practical use cases such as real‑time ETL, CTR calculation, and tag import.

ETLFlinkReal-time Data Warehouse
0 likes · 18 min read
How OPPO Built a Real‑Time Data Warehouse with Flink SQL
DataFunTalk
DataFunTalk
Jan 10, 2020 · Big Data

Design and Evolution of iQIYI's Real-Time Analytics Platform (RAP)

The article details iQIYI's Real-Time Analysis Platform (RAP), describing its motivation, architecture evolution from RAP 1.x to 2.x, OLAP engine selection, product design workflow, integration of Druid KIS and Flink, enhanced diagnostics, and real-world applications in membership monitoring, recommendation evaluation, and smart TV alerting.

DruidFlinkOLAP
0 likes · 12 min read
Design and Evolution of iQIYI's Real-Time Analytics Platform (RAP)
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 10, 2020 · Big Data

Async I/O for Dimension Table Joins in Apache Flink

This article explains how to handle dimension table joins in Apache Flink streaming by leveraging Async I/O to perform non‑blocking external lookups, provides detailed code examples for both synchronous and asynchronous functions, discusses configuration parameters, and outlines best practices and pitfalls.

Dimension Table JoinFlinkStreaming
0 likes · 16 min read
Async I/O for Dimension Table Joins in Apache Flink
iQIYI Technical Product Team
iQIYI Technical Product Team
Jan 9, 2020 · Big Data

Design and Evolution of iQIYI Real-Time Analysis Platform (RAP)

iQIYI’s Real‑Time Analysis Platform (RAP) combines Apache Druid with Spark/Flink to deliver minute‑level, low‑latency multidimensional analytics via a web wizard, supporting hundreds of streaming tasks and thousands of reports across membership, recommendation, and TV monitoring, while simplifying development and maintenance.

Apache DruidFlinkOLAP
0 likes · 13 min read
Design and Evolution of iQIYI Real-Time Analysis Platform (RAP)
dbaplus Community
dbaplus Community
Jan 6, 2020 · Big Data

How 58.com Built a Scalable Flink‑Based Real‑Time Data Platform (Wstream)

The article details how 58.com designed and evolved its one‑stop real‑time computation platform Wstream, migrating from Storm and Spark Streaming to Apache Flink, and describes the architecture, task isolation, stream‑SQL features, monitoring, and ongoing optimizations that enable processing of over 600 billion records daily.

FlinkStream SQLTask Migration
0 likes · 12 min read
How 58.com Built a Scalable Flink‑Based Real‑Time Data Platform (Wstream)
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 25, 2019 · Big Data

Understanding Flink StreamPartitioner and Its Implementations

Flink’s StreamPartitioner abstracts data routing in DataStream, offering eight built‑in partitioners—including Global, Shuffle, Rebalance, KeyGroup, Broadcast, Rescale, Forward, and Custom—each with distinct channel selection logic, illustrated with source code snippets and explanations of their runtime behavior.

DataStreamFlinkPartitioning
0 likes · 8 min read
Understanding Flink StreamPartitioner and Its Implementations
Qunar Tech Salon
Qunar Tech Salon
Dec 20, 2019 · Big Data

Understanding Flink Cluster Startup and Job Execution Process

This article explains the architecture of a Flink cluster, detailing the startup procedures for JobManager and TaskManager, the three deployment modes, and the end‑to‑end flow of a Flink job from client code through StreamGraph, JobGraph, ExecutionGraph to the physical execution on TaskManagers.

FlinkJobManagerTaskManager
0 likes · 10 min read
Understanding Flink Cluster Startup and Job Execution Process
Efficient Ops
Efficient Ops
Dec 17, 2019 · Operations

How Alibaba Scales Flink: Lessons in Big Data Operations

This article details Alibaba's massive Flink deployment, covering its historical background, the operational challenges of managing tens of thousands of nodes, the design of a comprehensive Flink management platform, and the automated solutions for fault handling, resource allocation, and performance testing in a large‑scale big‑data environment.

FlinkResource Allocationautomation
0 likes · 20 min read
How Alibaba Scales Flink: Lessons in Big Data Operations
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 17, 2019 · Big Data

Understanding Flink Sliding Windows and Performance Optimizations

This article explains Flink's sliding window mechanism, shows how the WindowAssigner and WindowOperator work with code examples, analyzes the performance impact of fine‑grained sliding windows, and proposes a practical workaround using tumbling windows combined with external storage such as Redis for efficient PV/UV aggregation.

FlinkPerformance OptimizationRedis
0 likes · 8 min read
Understanding Flink Sliding Windows and Performance Optimizations
Java Captain
Java Captain
Dec 17, 2019 · Backend Development

Top 10 Most Popular Java Open‑Source Projects on GitHub in November

This article lists and briefly describes the ten most starred Java open‑source projects on GitHub for November, covering tools such as NLP libraries, learning guides, big‑data frameworks, rapid‑development platforms, algorithm collections, job schedulers, code‑style checkers, traffic‑control systems, the Spring framework, and service‑discovery solutions.

FlinkFrameworksGitHub
0 likes · 5 min read
Top 10 Most Popular Java Open‑Source Projects on GitHub in November
Alibaba Cloud Developer
Alibaba Cloud Developer
Dec 5, 2019 · Artificial Intelligence

How Alibaba’s Alink Empowers Real‑Time Machine Learning on Flink

Alink, Alibaba’s open‑source machine‑learning platform built on Apache Flink, offers a rich library of batch and streaming algorithms, a Python API, iterative computation optimizations, and real‑world case studies, positioning it as a powerful AI solution for large‑scale, low‑latency data processing.

AIAlinkFlink
0 likes · 13 min read
How Alibaba’s Alink Empowers Real‑Time Machine Learning on Flink
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 4, 2019 · Big Data

Comprehensive Flink Interview Guide: Core Concepts, Advanced Topics, and Source‑Code Insights

This article provides an in‑depth Flink interview guide covering the framework’s core concepts, advanced features such as fault‑tolerance, state management, and checkpointing, as well as detailed explanations of its architecture, APIs, partitioning strategies, and source‑code flow, complete with code examples.

Flinkbig datadistributed systems
0 likes · 29 min read
Comprehensive Flink Interview Guide: Core Concepts, Advanced Topics, and Source‑Code Insights
Big Data Technology & Architecture
Big Data Technology & Architecture
Dec 1, 2019 · Big Data

Understanding Flink LatencyMarker: End-to-End Delay Measurement and Implementation Details

This article explains the background, source‑code analysis, and practical implementation of Flink's LatencyMarker feature for measuring end‑to‑end job latency, including metric exposure, configuration options, and code snippets illustrating how latency markers are emitted and processed within the streaming pipeline.

End-to-End LatencyFlinkLatencyMarker
0 likes · 6 min read
Understanding Flink LatencyMarker: End-to-End Delay Measurement and Implementation Details
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 29, 2019 · Big Data

Understanding Flink's Memory Management and Data Flow Architecture

This article explains how Flink manages memory through its MemorySegment abstraction, the implementations of HeapMemorySegment and HybridMemorySegment, the role of ByteBuffer, NetworkBufferPool and LocalBufferPool, and details the end‑to‑end data flow from RecordWriter to Netty transport, including key code snippets.

FlinkMemory ManagementNetwork Buffers
0 likes · 16 min read
Understanding Flink's Memory Management and Data Flow Architecture
21CTO
21CTO
Nov 27, 2019 · Big Data

How Xiaohongshu Scales Real‑Time Personalized Recommendations with Flink

The article summarizes Guo Yi’s 2019 Alibaba Cloud conference talk, outlining Xiaohongshu’s personalized recommendation architecture, detailing the data stack from ingestion to warehouse, and showcasing a Flink‑based real‑time multi‑dimensional user behavior aggregation use case, followed by a vision for the next year’s data architecture evolution.

Flinkdata architecturepersonalization
0 likes · 3 min read
How Xiaohongshu Scales Real‑Time Personalized Recommendations with Flink
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 26, 2019 · Big Data

Understanding Flink SQL Window Functions: Types, Implementation, and Emit Triggers

This article provides a comprehensive overview of Flink SQL window functions, detailing time‑based window types, their underlying implementation in the StreamExecGroupWindowAggregate operator, the processing flow of WindowOperator, timer handling, emit/trigger strategies, and practical code examples for Tumble, Hop, and Session windows.

EmitFlinkSQL
0 likes · 20 min read
Understanding Flink SQL Window Functions: Types, Implementation, and Emit Triggers
G7 EasyFlow Tech Circle
G7 EasyFlow Tech Circle
Nov 21, 2019 · Big Data

How G7 Combines AI, Big Data, and IoT to Transform Logistics

This article presents a detailed overview of G7's AI‑plus‑Big‑Data‑plus‑IoT platform for logistics, describing its neutral open architecture, real‑time data pipelines using Kafka and Flink, Lambda‑style storage in HBase/Hive, and the resulting safety‑insurance and analytics capabilities.

AIFlinkIoT
0 likes · 10 min read
How G7 Combines AI, Big Data, and IoT to Transform Logistics
Xianyu Technology
Xianyu Technology
Nov 21, 2019 · Big Data

Event-Driven Rule Engine for User Growth at Xianyu

To accelerate growth on Xianyu’s 20 million‑DAU platform, the team built an event‑driven rule engine with a SQL‑like DSL that translates user‑behavior streams into real‑time Flink/Blink queries, cutting rule development from four days to half a day and achieving sub‑5‑second processing latency.

DSLEvent StreamFlink
0 likes · 9 min read
Event-Driven Rule Engine for User Growth at Xianyu
58 Tech
58 Tech
Nov 20, 2019 · Big Data

Evolution of 58.com Real-Time Computing Platform and the One‑Stop Streaming Platform Wstream Built on Flink

This article details the technical evolution of 58.com’s real‑time computing platform, describing the shift from Storm and Spark Streaming to Apache Flink, the design of the one‑stop Wstream platform, its large‑scale deployment, stability measures, SQL streaming capabilities, task migration, diagnostics, optimizations, and future plans.

FlinkTask MigrationWstream
0 likes · 11 min read
Evolution of 58.com Real-Time Computing Platform and the One‑Stop Streaming Platform Wstream Built on Flink
DataFunTalk
DataFunTalk
Nov 19, 2019 · Big Data

Comprehensive Overview of Data Warehouses: Concepts, Evolution, Architecture, and Real‑time vs Offline Practices

This article provides a thorough introduction to data warehouses, traces their evolution, explains construction methodologies, compares offline, Lambda, and Kappa architectures, and presents real‑time warehouse case studies from Alibaba, Meituan, Xiaomi, Netflix, and OPPO, highlighting practical implementation details and challenges.

ETLFlinkKappa Architecture
0 likes · 14 min read
Comprehensive Overview of Data Warehouses: Concepts, Evolution, Architecture, and Real‑time vs Offline Practices
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 14, 2019 · Big Data

Comparison of Flink and Spark Structured Streaming: Joins, State Management, Fault Tolerance, and Backpressure

This article compares Flink and Spark Structured Streaming, detailing their differences in join capabilities, state management, fault‑tolerance mechanisms, exactly‑once semantics, back‑pressure handling, and table registration, while providing code examples and practical insights for real‑time big‑data processing.

FlinkJOINSpark Structured Streaming
0 likes · 13 min read
Comparison of Flink and Spark Structured Streaming: Joins, State Management, Fault Tolerance, and Backpressure
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 9, 2019 · Big Data

Comparative Study of Apache Flink and Spark Streaming at Xiaomi: Architecture, Performance, and Serialization

This article examines Xiaomi's migration from Spark Streaming to Apache Flink, comparing scheduling strategies, mini‑batch versus true streaming, resource utilization, latency, and serialization mechanisms, and concludes with practical insights and custom optimization techniques for large‑scale data processing.

FlinkMini-BatchResource Optimization
0 likes · 17 min read
Comparative Study of Apache Flink and Spark Streaming at Xiaomi: Architecture, Performance, and Serialization
Big Data Technology & Architecture
Big Data Technology & Architecture
Nov 7, 2019 · Big Data

Real‑time Dashboard with Flink: Streaming Order Data, Site Metrics, and Top‑N Merchandise Rankings

This article demonstrates how to build a one‑second‑refresh real‑time dashboard for e‑commerce order data using Apache Flink, Kafka, and Redis, covering JSON message parsing, processing‑time windows, stateful aggregation for site‑level KPIs, and efficient top‑N product ranking via Redis sorted sets.

FlinkKafkaRedis
0 likes · 11 min read
Real‑time Dashboard with Flink: Streaming Order Data, Site Metrics, and Top‑N Merchandise Rankings
DataFunTalk
DataFunTalk
Nov 7, 2019 · Big Data

Real-Time Computing Engine at Beike: Architecture, Practices, and Future Plans

This article details Beike's real‑time computing engine, covering its background, streaming platform built on Spark Streaming and Flink, data ingestion via Kafka, metadata handling, SQL‑based task development, monitoring, storage solutions, and future roadmap for resource management and AI‑enhanced monitoring.

Data PipelineFlinkKafka
0 likes · 14 min read
Real-Time Computing Engine at Beike: Architecture, Practices, and Future Plans
dbaplus Community
dbaplus Community
Oct 22, 2019 · Big Data

How Weibo Built a Billion‑Log Real‑Time Data Platform with Flink

This article details how Weibo’s advertising team designed and implemented a real‑time data platform capable of processing over a hundred billion daily logs, covering technology selection, Flink advantages, architecture evolution, data processing pipelines, component libraries, fault‑tolerance strategies, and the construction of a multi‑layer real‑time data warehouse.

CheckpointFlinkbig data
0 likes · 25 min read
How Weibo Built a Billion‑Log Real‑Time Data Platform with Flink
Xianyu Technology
Xianyu Technology
Oct 16, 2019 · Big Data

Xianyu's Complex Event Processing (CEP) System Design and Implementation

Xianyu’s Complex Event Processing system, built on Alibaba’s Blink (Flink) and a custom SQL‑like DSL, standardizes event I/O, lets users define sequence, window and aggregation rules, and combines an interactive rule service, SLS source, parser, job manager and MetaQ sink to achieve ~100 k QPS, sub‑second latency, fault‑tolerant, and rule‑to‑production turnaround in about thirty minutes.

CEPDSLFlink
0 likes · 9 min read
Xianyu's Complex Event Processing (CEP) System Design and Implementation
JD Retail Technology
JD Retail Technology
Oct 14, 2019 · Databases

Overview of JDNoSQL Platform and Its Real-Time Advertising Use Cases

The article introduces JDNoSQL, a distributed column‑oriented key‑value store built on HDFS, outlines its core features, describes various business scenarios including real‑time ad computation, details the system architecture with Kafka and Flink, and presents table designs for ad impression and click statistics.

FlinkKafkaNoSQL
0 likes · 13 min read
Overview of JDNoSQL Platform and Its Real-Time Advertising Use Cases
UCloud Tech
UCloud Tech
Oct 11, 2019 · Big Data

Real‑Time Student Performance Analytics with Flink and Spark

This article demonstrates how to build a real‑time education analytics system by streaming answer data through Kafka into Flink or Spark, performing per‑question, per‑grade, and per‑subject aggregations, and optionally accelerating development with UFlink SQL.

Education AnalyticsFlinkKafka
0 likes · 17 min read
Real‑Time Student Performance Analytics with Flink and Spark
58 Tech
58 Tech
Oct 10, 2019 · Big Data

Optimizing Real‑Time Feature Extraction at 58.com: Migrating from Spark Streaming to Flink

This article describes how 58.com’s commercial engineering team redesigned its real‑time feature‑mining pipeline—replacing a minute‑level Spark Streaming framework with Flink—to achieve sub‑second latency, higher throughput, stronger fault‑tolerance, and end‑to‑end exactly‑once semantics for user‑profile generation in the second‑hand‑car recommendation scenario.

Exactly-OnceFlinkSpark
0 likes · 14 min read
Optimizing Real‑Time Feature Extraction at 58.com: Migrating from Spark Streaming to Flink
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 9, 2019 · Big Data

Choosing and Using Flink State Backends: MemoryStateBackend, FsStateBackend, and RocksDBStateBackend

This article explains how Flink checkpoints persist state, compares the three built‑in state backends (MemoryStateBackend, FsStateBackend, RocksDBStateBackend), discusses their configurations, advantages, limitations, and provides guidance on selecting the appropriate backend for different big‑data streaming scenarios.

CheckpointFlinkMemoryStateBackend
0 likes · 10 min read
Choosing and Using Flink State Backends: MemoryStateBackend, FsStateBackend, and RocksDBStateBackend
HomeTech
HomeTech
Oct 9, 2019 · Big Data

Design and Implementation of a Flink‑Based Real‑Time Data Platform at Autohome

This article describes how Autohome migrated its real‑time analytics from Storm to a Flink‑SQL platform, detailing the architectural design, development and operational advantages, practical use cases such as recommendation metrics, and future plans for ecosystem expansion and open‑source release.

FlinkSQLdata warehouse
0 likes · 12 min read
Design and Implementation of a Flink‑Based Real‑Time Data Platform at Autohome
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 21, 2019 · Big Data

Deploying Apache Flink on Kubernetes: A Step‑by‑Step Guide

This tutorial explains how to run Apache Flink jobs on Kubernetes by building Docker images, deploying JobManager and TaskManager components with Kubernetes manifests, configuring high‑availability with ZooKeeper and HDFS, and using SavePoints and scaling techniques to manage and extend Flink streaming applications.

DockerFlinkJobManager
0 likes · 14 min read
Deploying Apache Flink on Kubernetes: A Step‑by‑Step Guide
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 16, 2019 · Big Data

Comprehensive Flink Interview Guide: Architecture, APIs, Operators, and Advanced Topics

This guide provides a detailed overview of Apache Flink covering its core streaming engine, APIs (DataSet, DataStream, Table), architectural components, comparison with Spark Streaming, partitioning, parallelism, restart strategies, state backends, time semantics, watermarks, SQL processing, fault‑tolerance mechanisms, memory management, serialization, RPC framework, back‑pressure handling, operator chaining, and practical tips for interview preparation.

Apache FlinkDataFlowFlink
0 likes · 22 min read
Comprehensive Flink Interview Guide: Architecture, APIs, Operators, and Advanced Topics
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 11, 2019 · Big Data

Evolution of Zhihu's Real-Time Data Warehouse: From Spark Streaming 1.0 to Flink‑Based 2.0

This article details Zhihu's real‑time data warehouse evolution, describing the 1.0 Spark Streaming architecture, its limitations, and the 2.0 redesign that introduces Flink, layered data models, streaming and batch ETL, metric storage choices, and future roadmap for scalable, low‑latency analytics.

FlinkLambda ArchitectureReal-time Data Warehouse
0 likes · 19 min read
Evolution of Zhihu's Real-Time Data Warehouse: From Spark Streaming 1.0 to Flink‑Based 2.0
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 6, 2019 · Big Data

Big Data Development Interview Guide and Skill Tree Overview

This article provides a comprehensive interview roadmap for big data developers, outlining essential Java fundamentals, JVM internals, Linux basics, distributed theory, core frameworks such as Hadoop, Spark, Flink, Kafka, Netty, HBase, Hive, and practical algorithm topics, while also offering resume and career advice for aspiring candidates.

FlinkHadoopKafka
0 likes · 15 min read
Big Data Development Interview Guide and Skill Tree Overview
Xueersi Online School Tech Team
Xueersi Online School Tech Team
Sep 6, 2019 · Big Data

Real-Time Data Architecture, Evolution, and Applications at an Online School

The article details the six‑layer big‑data architecture of an online school, chronicles its migration from Storm to Spark Streaming and finally to Flink, and showcases concrete real‑time applications such as gateway monitoring, user‑profile tagging, renewal reporting, and advertising analysis, while outlining future development directions.

AnalyticsBig Data ArchitectureData Pipeline
0 likes · 14 min read
Real-Time Data Architecture, Evolution, and Applications at an Online School
Big Data Technology & Architecture
Big Data Technology & Architecture
Sep 5, 2019 · Big Data

Applying Flink CEP for Complex Event Processing at Haolo Mobility

This article explains how Flink CEP, a complex event processing library for Apache Flink, is employed at Haolo Mobility to detect intricate patterns in endless data streams by modeling patterns as states and using pattern conditions for state transitions, illustrating its practical application in real‑world big‑data scenarios.

CEPFlinkHaolo Mobility
0 likes · 2 min read
Applying Flink CEP for Complex Event Processing at Haolo Mobility
Tongcheng Travel Technology Center
Tongcheng Travel Technology Center
Sep 3, 2019 · Big Data

Practical Experiences and Lessons Learned in Building a Flink‑Based Real‑Time Computing Platform at Tongcheng‑Elong

This article details the design, implementation, and optimization of a Flink‑based real‑time computing platform at Tongcheng‑Elong, covering the evolution from Storm to Flink, support for FlinkSQL and FlinkStream, metric collection, logging, data lineage, savepoint management, and numerous stability fixes contributed back to the open‑source community.

FlinkReal-Time ComputingStreaming
0 likes · 16 min read
Practical Experiences and Lessons Learned in Building a Flink‑Based Real‑Time Computing Platform at Tongcheng‑Elong