Exploring Open Source Big Data and AI Technologies
The Fluss project, an open‑source next‑generation stream storage engine donated by Alibaba, has entered the Apache Software Foundation incubator, offering columnar streaming, real‑time updates, lake‑flow integration, impressive performance metrics, and a growing global developer community.
Kafka 4.0 marks a milestone release that replaces ZooKeeper with the KRaft consensus engine, improves scalability and performance, introduces a server‑side consumer‑group protocol, adds shared‑group queue capabilities, and updates Java requirements and documentation, delivering a more robust and flexible streaming platform.
This article explains the three main types of computing power—general (通算), intelligent (智算), and supercomputing (超算)—detailing their definitions, typical processor architectures, and real‑world application scenarios across everyday office tasks, AI workloads, and large‑scale scientific research.
This article explains CDC fundamentals, details Flink CDC's architecture and advantages, provides setup steps, code examples for SQL and DataStream APIs, discusses performance tuning, consistency, common issues, and typical real‑time data integration scenarios.
This article explains how to replicate Deepseek RI's slow‑thinking inference using the GRPO reinforcement‑learning algorithm on the Qwen2.5‑7B model in a free Colab notebook, covering the underlying COT concept, reward‑function design, data preparation, training configuration, and observed results.
This article shows how AI tools like DeepSeek can dramatically speed up data‑engineering tasks—such as fixing long‑running SQL queries, building real‑time data pipelines with Flink, and deciphering legacy stored procedures—while offering concrete prompts, real‑world case studies, and five time‑saving techniques.
This guide explains why DeepSeek is a valuable AI assistant, outlines the challenges of high demand, and provides step‑by‑step instructions for creating, configuring, testing, and publishing a DeepSeek AI assistant within the DingTalk platform to ensure stable access.
The article describes how NetEase News' recommendation team replaced a slow, batch‑oriented data‑warehouse pipeline with a Flink‑based, Apache Paimon real‑time wide‑table solution that supports partial updates, reduces latency from hours to minutes, and lowers processing costs while handling both deduplication and non‑deduplication recommendation scenarios.
This article demonstrates, with step‑by‑step examples, how to capture MySQL changes via Flink CDC and stream them in real time into Apache Doris using the Doris Flink Connector, covering CDC concepts, connector features, environment setup, SQL client usage, and data verification.
The talk outlines the evolution of Alibaba Cloud's open‑source big data platform from Hadoop‑based EMR to a 3.0 architecture featuring a streaming lakehouse, full serverless compute and storage, AI‑driven operations, and upcoming vector search services, highlighting technical motivations, challenges, and product releases.
