BestHub
Sign in
Big Data Technology & Architecture
Author

Big Data Technology & Architecture

Wang Zhiwu, a big data expert, dedicated to sharing big data technology.

1.0k
Articles
0
Likes
4.6k
Views
0
Comments
Recent Articles

Latest from Big Data Technology & Architecture

100 recent articles max
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 24, 2025 · Big Data

Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More

Apache Kafka 4.0 introduces a groundbreaking KRaft architecture that removes ZooKeeper, a revamped consumer‑group protocol that dramatically speeds up rebalancing, a new queue mode for point‑to‑point messaging, upgraded Java requirements, streamlined APIs, and numerous performance and security enhancements, reshaping both development and operations for large‑scale streaming workloads.

Java 11Java 17KRaft
0 likes · 12 min read
Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 17, 2025 · Big Data

Lakehouse Implementations at Leading Companies: Challenges, Solutions, and Benefits

This article reviews how major tech firms such as Alibaba, Tencent, ByteDance, and Kuaishou tackled lakehouse challenges—including architecture fragmentation, cost, scalability, and complex multimodal data—by adopting real‑time lakehouse solutions like Flink + Paimon, Iceberg + StarRocks, Hudi + LAS, and Doris + Alluxio, and outlines the resulting performance and cost gains.

DorisFlinkLakehouse
0 likes · 9 min read
Lakehouse Implementations at Leading Companies: Challenges, Solutions, and Benefits
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 3, 2025 · Big Data

The Turning Point for Data Development: From Traditional Data Engineering to AI Data Engineering

The article analyzes how the rapid rise of open‑source large‑model AI in 2025 is reshaping the data development profession, urging developers to transition from specialized data‑engineer roles to full‑stack AI data engineering skills such as distributed computing, lake‑house architectures, and model tuning.

AIData EngineeringDistributed Computing
0 likes · 7 min read
The Turning Point for Data Development: From Traditional Data Engineering to AI Data Engineering
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 4, 2025 · Artificial Intelligence

How Large Language Models Are Transforming Data Development and Developer Roles

The article discusses how large language model tools such as Cursor, DeepSeek, and Doubao are increasingly assisting code writing, SQL translation, job‑failure analysis, and documentation in data‑development workflows, while also reshaping job requirements and creating new opportunities for skilled developers.

AIData developmentSQL automation
0 likes · 5 min read
How Large Language Models Are Transforming Data Development and Developer Roles
Big Data Technology & Architecture
Big Data Technology & Architecture
Feb 1, 2025 · Big Data

Douyin Group Data Asset Management Platform: Comprehensive Data Lineage Overview and Practices

This article presents a detailed overview of Douyin Group's Data Asset Management Platform, focusing on the evolution, architecture, modeling, metrics, and application scenarios of its large‑scale data lineage system, and outlines future directions for full‑coverage, fine‑grained lineage capabilities.

Data Asset Managementbig datadata lineage
0 likes · 17 min read
Douyin Group Data Asset Management Platform: Comprehensive Data Lineage Overview and Practices
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 15, 2025 · Big Data

From Operations to Data Engineering: A Student’s Real‑World Journey and Practical Guide

This article shares a data‑engineering student’s personal experience—from a misaligned operations role to mastering big‑data technologies, building a portfolio, crafting a targeted resume, and navigating multi‑stage interviews—offering concrete advice and a structured learning roadmap for aspiring data professionals.

Data EngineeringInterview PreparationLearning Path
0 likes · 14 min read
From Operations to Data Engineering: A Student’s Real‑World Journey and Practical Guide
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 13, 2025 · Big Data

How Apache Paimon Manages Snapshot Expiration: Synchronous vs Asynchronous Modes

This article explains Apache Paimon's snapshot expiration mechanism, comparing synchronous and asynchronous execution modes, their advantages and drawbacks, and how table properties control expiration to balance data consistency, performance, and back‑pressure in large‑scale data processing systems.

Apache PaimonSynchronousdata consistency
0 likes · 6 min read
How Apache Paimon Manages Snapshot Expiration: Synchronous vs Asynchronous Modes
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 2, 2025 · Big Data

Apache Paimon: Core Capabilities, Table Types, LSM Tree, Buckets, Merge Engines, and Operational Details

This article provides a comprehensive overview of Apache Paimon, covering its real‑time lake ingestion, unified stream‑batch processing, table types (primary‑key and append‑only), LSM‑tree storage, bucket mechanisms, merge‑engine options, compaction strategies, concurrency control, consumption methods, tag management, data cleanup, and system tables for big‑data workloads.

Apache PaimonFlinkLSM-tree
0 likes · 25 min read
Apache Paimon: Core Capabilities, Table Types, LSM Tree, Buckets, Merge Engines, and Operational Details