Sharing cutting‑edge big data and AI technologies, with occasional lifestyle insights.
The article outlines a comprehensive data‑warehouse quality assurance framework covering data lineage, layer‑level testing (timeliness, completeness, accuracy), application‑level checks, downgrade and backup strategies, and future plans for automated data shape reporting and static SQL analysis.
The article explains the concepts of data lakes and lakehouses, compares them with traditional data warehouses, outlines the reliability, performance, and security challenges of data lakes, and then details a practical lakehouse implementation using Apache Iceberg, Flink SQL, CDC pipelines, and supporting tools such as Hive Metastore and Trino.
The article analyzes common pain points in data‑warehouse metric management from business, technical, and product perspectives, outlines unified goals, defines a comprehensive model—including data domains, dimensions, atomic, derived, and composite metrics—and presents a step‑by‑step process for designing, implementing, and governing a scalable metric system.
This article explains the fundamentals of network flow control, compares TCP‑based and credit‑based backpressure in Flink, walks through buffer pool interactions, and shows why dynamic backpressure cannot fully replace static rate limiting in real‑world streaming pipelines.
The article examines OpenAI’s latest DevDay announcements—AgentKit, the GA version of Codex, the Apps SDK, and the Sora 2 API—detailing their technical breakthroughs, core modules, and potential impact on AI agent development, software engineering, and multimodal content creation.
This guide walks through adding the Elasticsearch sink to a Flink job, configuring connection and bulk settings, implementing custom failure handlers, and enabling checkpoint‑based retry mechanisms to ensure reliable streaming writes.
The article outlines the problems of early data services, proposes a two‑step data warehouse solution—abstracting a data model and extracting, cleaning, and storing data from MySQL, MongoDB, and Elasticsearch—then details storage selection, dimensional modeling, incremental ETL, and Airflow workflow management.
The article presents a three‑step methodology—identifying core requirements, constructing a capability‑matching matrix, and validating through pilot scenarios—to help enterprises evaluate data warehouse modeling tools based on scale, architecture compatibility, team skills, automation, extensibility, cost, performance testing, and continuous optimization.
The article details Zhihu's step‑by‑step evolution from a Spark‑Streaming‑based real‑time data warehouse (v1.0) to a Flink‑Streaming SQL‑driven platform (v2.0), explaining design choices, Lambda architecture implementation, scalability challenges, and future automation plans.
This article details practical Spark performance optimizations—including RDD reuse, broadcast variables, Kryo serialization, parallelism settings, shuffle tuning, and JVM tweaks—while also presenting systematic solutions for data skew, shuffle failures, serialization errors, and YARN mode issues, all illustrated with concrete code snippets and examples.
