Author

Big Data Technology & Architecture

Wang Zhiwu, a big data expert, dedicated to sharing big data technology.

1.0k

Articles

Likes

Views

Comments

Latest from Big Data Technology & Architecture

100 recent articles max

Big Data Technology & Architecture

Nov 27, 2025 · Artificial Intelligence

Rule‑Based NLQ vs LLMs: How ChatBI’s MQL Engine Delivers Precise BI Queries

The article explains how the rule‑based NLQ component of ChatBI replaces large language models with a detailed dictionary‑driven architecture, using a custom Metrics Query Language (MQL) to transform natural‑language business questions into accurate SQL, highlighting its stability, low cost, transparency, and limitations compared to LLM solutions.

Data QueryLLM comparisonMQL

0 likes · 12 min read

Rule‑Based NLQ vs LLMs: How ChatBI’s MQL Engine Delivers Precise BI Queries

Big Data Technology & Architecture

Nov 17, 2025 · Big Data

Flink 2025 Updates: Disaggregated State, AI Agents, and SQL Enhancements

The 2025 Flink release introduces a disaggregated state management architecture for cloud‑native elasticity, AI‑driven Flink Agents with LLM, Memory and Tool support, Delta Join and VARIANT type for semi‑structured data, adaptive batch execution, incremental checkpoints, high‑speed network optimizations, and new SQL and Process Table Functions, reshaping real‑time analytics.

Disaggregated StateFlinkReal-time analytics

0 likes · 8 min read

Flink 2025 Updates: Disaggregated State, AI Agents, and SQL Enhancements

Big Data Technology & Architecture

Nov 3, 2025 · Big Data

Taming Small Files in Paimon: Proven Tuning Strategies for Better Performance

This article explains how small‑file issues in Paimon's streaming data lake architecture degrade system stability and query speed, and presents practical parameter‑tuning, table‑level settings, asynchronous compaction, and monitoring techniques to mitigate those problems.

FlinkPaimonSmall Files

0 likes · 7 min read

Taming Small Files in Paimon: Proven Tuning Strategies for Better Performance

Big Data Technology & Architecture

Oct 30, 2025 · Backend Development

What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained

Apache Kafka 4.1.0 introduces native queue semantics, a new Streams rebalancing protocol, multi‑version Connect plugins, a revamped consumer‑group protocol, enhanced transaction safety, and numerous client, monitoring, and security improvements, offering a comprehensive upgrade over the 4.0 release.

KafkaMessage QueuesStreaming

0 likes · 6 min read

What’s New in Apache Kafka 4.1? Core Features and Architecture Changes Explained

Big Data Technology & Architecture

Oct 20, 2025 · Big Data

Unlocking Lakehouse Power: Paimon and Doris Integrated Solutions

This article reviews how Paimon and Doris combine to solve unified storage, data visibility, and performance challenges in modern lakehouse architectures, detailing their complementary features, integration capabilities, and real‑world use cases from leading companies.

AnalyticsPaimonbig data

0 likes · 8 min read

Unlocking Lakehouse Power: Paimon and Doris Integrated Solutions

Big Data Technology & Architecture

Oct 13, 2025 · Databases

Apache Doris 3.1 Unveiled: Variant, Index, and Lakehouse Boosts

The Apache Doris 3.1 release strengthens lake‑house capabilities with major upgrades to the VARIANT data type, vertical compaction, inverted index storage, new tokenizers, enhanced materialized view support for Iceberg/Paimon/Hudi, and numerous query‑performance optimizations such as faster partition pruning and dynamic partition clipping, offering smoother handling of thousands of columns and large‑scale semi‑structured data.

Apache DorisQuery PerformanceVARIANT

0 likes · 8 min read

Apache Doris 3.1 Unveiled: Variant, Index, and Lakehouse Boosts

Big Data Technology & Architecture

Oct 10, 2025 · Artificial Intelligence

Explore Cutting-Edge Open-Source AI Projects: Gemini CLI, AI Engineering Hub, and GPT‑5 Demos

This article introduces several noteworthy open‑source AI projects—including Google’s Gemini CLI, the AI‑Engineering‑Hub learning repository, and OpenAI’s GPT‑5 coding examples—providing URLs, key features, and visual previews to help developers quickly explore and adopt cutting‑edge AI tools.

AI Engineering HubArtificial IntelligenceGPT-5

0 likes · 3 min read

Explore Cutting-Edge Open-Source AI Projects: Gemini CLI, AI Engineering Hub, and GPT‑5 Demos

Big Data Technology & Architecture

Sep 24, 2025 · Big Data

Avoid These 6 Common Paimon Data Loss Pitfalls in Flink and Spark

Learn the six typical scenarios that cause data loss when writing to Paimon—ranging from checkpoint failures and misconfigured partial‑update mode to incorrect sequence fields, snapshot retention issues, concurrent bucket writes, and outdated Spark versions—and how to prevent each problem.

CheckpointFlinkPaimon

0 likes · 5 min read

Avoid These 6 Common Paimon Data Loss Pitfalls in Flink and Spark

Big Data Technology & Architecture

Sep 17, 2025 · Big Data

How I Landed a Big Data Job in a Small City: My Real‑World Journey

A former communications engineer shares his step‑by‑step transition into big data, detailing study challenges, resume crafting, interview strategies, and the lessons learned from navigating job offers in both small towns and major tech firms.

Career transitionbig data jobinterview strategy

0 likes · 8 min read

How I Landed a Big Data Job in a Small City: My Real‑World Journey

Big Data Technology & Architecture

Sep 10, 2025 · Databases

When to Use Materialized Views in Production: Benefits, Types, and Pitfalls

This article explains what materialized views are, outlines their advantages such as query acceleration, lightweight ETL, and lake‑warehouse integration, classifies them by sync mode, table count, and refresh strategy, and highlights their limitations and best‑practice recommendations for production use.

Data WarehousingDatabase PerformanceETL

0 likes · 6 min read

When to Use Materialized Views in Production: Benefits, Types, and Pitfalls