Wang Zhiwu, a big data expert, dedicated to sharing big data technology.
This article provides a comprehensive technical deep dive into Ray Data, covering its architecture, streaming execution model, shuffle implementation, fault tolerance, performance tuning, and best practices for AI data pipelines, positioning it as a superior alternative to Spark for GPU-intensive workloads.
This article introduces Daft, Ray, and Lance as the core layers of modern AI data infrastructure: Daft handles multimodal data computation, Ray manages distributed scheduling across heterogeneous resources, and Lance provides a lakehouse format optimized for vector search and random access, together replacing fragmented toolchains.
This article compares Snowflake's Semantic View and Databricks' Metric View as semantic layer implementations for AI agents, detailing their architectural differences, code examples, and best practices for modeling, deployment, and AI accuracy tuning.
Analysis of AI-era data engineering job descriptions reveals emerging technology stacks including RAG pipelines, model post-training, agent engineering, and AI infrastructure, outlining five key technical directions for data professionals transitioning into AI roles.
The article explains why traditional physical‑table‑plus‑SQL pipelines no longer meet AI‑driven data needs and proposes a semantic‑engineered approach—semantic assets, a governed semantic layer, logical tables, and agents—to make data meaning executable, verifiable, and governable.
The article examines advanced data‑warehouse RAG challenges, explains why pure vector retrieval falls short, details a BM25‑plus‑vector hybrid approach, showcases AI‑driven metric platforms and data‑agent design, and outlines practical factors for building a robust data‑agent foundation.
This article (part 1) presents twelve high‑level interview questions from top tech firms covering the shift from AI‑Ready to Data‑Agent‑Ready data warehouses, AI‑centric metadata and lineage, lakehouse formats like Paimon, and key Iceberg features that empower AI‑driven analytics.
The article examines Apache Ossie's role as an incubating Apache project that defines a vendor‑neutral, AI‑ready semantic metadata standard, addressing semantic islands, improving AI Agent reliability, and enabling reusable, exchangeable business semantics across analytics, BI and data platforms.
The article outlines a practical framework for assessing data‑centric AI agents, highlighting challenges such as nondeterminism, black‑box behavior, and error amplification, and proposes concrete dimensions—result correctness, semantic consistency, query quality, security, and explainability—to ensure zero‑tolerance accuracy and reproducibility.
The article analyzes the May 2026 releases of Apache Iceberg V3 and Spark 4.0, explaining how Deletion Vectors, the native VARIANT type, and Spark Connect together transform semi‑structured data handling and real‑time upserts in lakehouse architectures.
