Author

Big Data Technology & Architecture

Wang Zhiwu, a big data expert, dedicated to sharing big data technology.

1.0k

Articles

Likes

3.3k

Views

Comments

Latest from Big Data Technology & Architecture

100 recent articles max

Big Data Technology & Architecture

Jul 8, 2025 · Big Data

Flink’s AI Agents and Disaggregated State: Transforming Big Data

The article reviews key topics from the FFA2025 Singapore conference, highlighting Flink’s new AI‑focused Agents framework, the breakthrough Flink 2.0 disaggregated state architecture, emerging lake storage solutions like Paimon, and the Fluss streaming table store, illustrating how big‑data platforms are evolving for AI workloads.

AI AgentsBig DataData Lake

0 likes · 6 min read

Flink’s AI Agents and Disaggregated State: Transforming Big Data

Big Data Technology & Architecture

Jul 4, 2025 · Big Data

Spark 4.0: New Features, Performance Gains, and Why It Still Leads Big Data

Despite the hype around Flink and AI models, Spark 4.0’s release brings a lightweight Python client, Spark Connect GA, enhanced SQL optimization, vectorized execution, and AI integration, reaffirming its leading position in the big‑data ecosystem while hinting at future challenges and innovations.

Big DataData EngineeringPython

0 likes · 6 min read

Spark 4.0: New Features, Performance Gains, and Why It Still Leads Big Data

Big Data Technology & Architecture

Jul 1, 2025 · Big Data

What’s New in Apache Hive 4.0? Key Features and Industry Outlook

After a weekend dive into Apache Hive’s official Wiki and GitHub, this article highlights Hive’s declining visibility compared to Spark and Flink, examines its 4.0 release’s major features—including Iceberg integration, enhanced ACID, cost‑based optimizer upgrades, and Ozone support—while reflecting on its role in modern data ecosystems.

Apache HiveBig DataHive 4.0

0 likes · 4 min read

What’s New in Apache Hive 4.0? Key Features and Industry Outlook

Big Data Technology & Architecture

Jun 24, 2025 · Big Data

How Leading Companies Leverage Apache Paimon for Real‑Time Lakehouse Success

This article summarizes how major tech firms such as Vivo, Shopee, Alibaba, and TikTok adopt Apache Paimon to unify batch and streaming data pipelines, improve latency, reduce costs, and optimize storage, highlighting key challenges, architectural solutions, and real‑world performance gains.

PaimonReal-time databatch processing

0 likes · 8 min read

How Leading Companies Leverage Apache Paimon for Real‑Time Lakehouse Success

Big Data Technology & Architecture

Jun 20, 2025 · Big Data

2025 Mid-Year Data Job Market Trends: Skills, Hiring Preferences, and Career Advice

This semi‑annual report analyzes over 120 interview experiences to reveal current hiring trends, required skills, and strategic advice for data professionals targeting mid‑to‑senior positions in large‑scale companies during the first half of 2025.

2025 trendsCareer AdviceData Engineering

0 likes · 9 min read

2025 Mid-Year Data Job Market Trends: Skills, Hiring Preferences, and Career Advice

Big Data Technology & Architecture

Jun 11, 2025 · Big Data

How to Solve Common Paimon Performance Issues in Flink: Small Files, OOM, and More

This article compiles frequent problems encountered when using Paimon with Flink—such as small‑file generation, write‑performance bottlenecks, OOM/GC issues, file‑deletion conflicts, dimension‑table join slowness, and snapshot expiration—and provides practical configuration and optimization solutions.

Big DataFlinkOptimization

0 likes · 9 min read

How to Solve Common Paimon Performance Issues in Flink: Small Files, OOM, and More

Big Data Technology & Architecture

Jun 10, 2025 · Big Data

Transforming Real‑Time Analytics: Incremental Computing with Lakehouse Architecture

This article examines how Xiaohongshu replaced its costly Lambda architecture with a real‑time lakehouse built on Iceberg, Paimon, Spark, and StarRocks, achieving minute‑level latency, higher data quality, lower resource consumption, and dramatically faster query performance.

Big Data ArchitectureIcebergPaimon

0 likes · 7 min read

Transforming Real‑Time Analytics: Incremental Computing with Lakehouse Architecture

Big Data Technology & Architecture

Jun 9, 2025 · Databases

Why Data Warebase Could Be the Next Game‑Changer for AI Workloads

The article examines how emerging data‑infrastructure trends, multi‑modal databases like Neon, Supabase, and ClickHouse, and the convergence of OLTP, OLAP, and vector search are reshaping AI workloads, introducing the Data Warebase concept that unifies warehouse and database capabilities to meet modern AI workflow demands.

AIHTAPReal-time Analytics

0 likes · 32 min read

Why Data Warebase Could Be the Next Game‑Changer for AI Workloads

Big Data Technology & Architecture

Jun 6, 2025 · Big Data

How to Ace Big Data Interviews: A Complete 3‑Month Prep Guide

This guide outlines a step‑by‑step three‑month preparation plan—including resume building, project showcase, interview mindset, mock sessions, and offer negotiation—to help candidates secure high‑paying big‑data positions at top companies.

Big Datainterview preparationoffer negotiation

0 likes · 7 min read

How to Ace Big Data Interviews: A Complete 3‑Month Prep Guide

Big Data Technology & Architecture

Jun 5, 2025 · Big Data

Flink Web UI Monitoring and End‑to‑End Latency Implementation Guide

This article explains the key monitoring items of the Flink Web UI, details task topology, operator and system metrics, checkpoint and log inspection, and provides two practical solutions—custom metrics and distributed tracing—to measure and visualize full‑chain latency in Flink jobs.

Big DataFlinkMetrics

0 likes · 10 min read

Flink Web UI Monitoring and End‑to‑End Latency Implementation Guide