1 views collected around this technical thread.
This article shares Ant Group's practical experience and evolution of data architecture, covering theoretical foundations, analysis of current challenges, proposed upgrade solutions based on domain‑driven design, and future outlook through complex network theory to improve scalability, governance, and resilience of large‑scale data systems.
This article shares the foreign bank's big data implementation journey, covering background and goals, overall planning and architecture, practical insights, phased rollout, data governance, security, and Q&A, illustrating how a unified data platform, storage‑compute separation, and AI‑driven tools drive business innovation.
The article details the background, architectural analysis, multi‑tenant redesign, engine selection enhancements, compatibility adaptations, stability fixes, containerized deployment, performance optimizations, and measurable business outcomes of the Intelligent Acceleration Engine upgrade using Apache Kyuubi and StarRocks within the 58 big data platform.
This article shares Ant Group Data Technology’s comprehensive view on data governance, covering its concepts and framework, practical strategies such as architecture, standards, platforms and digital operations, real‑world implementations like distributed warehouses and the OneData system, and future trends involving AI and automation.
This article explains how Tencent Cloud's TBDS platform evolves its architecture by adopting Apache Gravitino as a unified metadata lake, detailing the challenges of legacy versus new lakehouse designs, storage and compute separation, unified data access, permission management, and the resulting benefits for big‑data and AI workloads.
The article traces the progression from the human DIKW information hierarchy to its computer‑world counterpart, illustrates how a homestay platform’s data architecture evolves through relational, NoSQL, search, and data‑warehouse layers, and introduces the next‑generation distributed Data Warebase that unifies structured, semi‑structured, and vectorized knowledge to meet modern AI‑driven business demands.
This article surveys the evolution of modern data technologies—from the 4V characteristics of big data and the limitations of traditional relational databases, through the rise of NoSQL and polyglot persistence, to embedding‑driven vector search, hybrid retrieval and RAG, illustrating how each paradigm frees applications from data constraints.
DataFunSummit2024, co-hosted by Bilibili, convenes industry experts, scholars, and enterprise leaders across six forums to discuss next‑generation data architecture, showcasing Bilibili’s Iceberg‑based stream‑batch innovations, AI‑BI analytics, NoETL practices, and emerging alternatives to Lambda architecture.
Amid growing data demands, this article explains the data lake technology maturity curve, detailing lake‑warehouse architectural patterns, design principles, core functionalities, and the four leading open‑source solutions (Hudi, Iceberg, Delta Lake, Paimon) to guide enterprises in building flexible, scalable, and governed data platforms.
The article chronicles the author’s 30‑year programming career, detailing early experiences, the evolution from JavaScript projects to large‑scale big‑data architectures, cloud migration, business‑agnostic framework design, interactive analytics, and reflections on becoming an independent software architect.
This article reviews the current state of data platforms, analyzes the third wave of data infrastructure evolution driven by databases, big data and generative AI, proposes next‑generation lakehouse and cloud‑native architectural directions, and outlines future trends and unresolved challenges for AI‑centric data platforms.
This article presents Qichacha's comprehensive data‑cost‑reduction strategy, detailing its Hadoop‑based three‑pillar architecture, layered data warehouse, Hive upgrades, unified metadata across multi‑cloud clusters, middleware choices such as Alluxio and JuiceFS, version‑compatible hybrid clouds, and Kubernetes‑driven resource orchestration to achieve scalable, low‑cost data processing.
This article explores the DIKW hierarchy, explains how data evolves into information, knowledge, and wisdom, examines traditional data models and products, critiques existing multi‑system architectures, and proposes a new distributed Data Warebase that unifies structured, semi‑structured, and vectorized knowledge to enable intelligent data-driven applications.
The forum presents a series of technical talks on multi‑cloud data architecture, covering Xiaomi’s lake‑warehouse practice, cross‑border e‑commerce data platforms, Alluxio‑based machine‑learning acceleration, Qichacha’s cost‑effective data solutions, and Kuaishou’s Flink on Kubernetes migration, highlighting strategies, implementations, and audience benefits.
The article presents a comprehensive case study of XinXuan Group’s data ecosystem, detailing its business background, data platform construction, governance framework, technical architecture, and practical analytics applications that support live‑commerce operations and strategic decision‑making.
This article explains the TOGAF framework and its 4A architecture model—Business, Technology, Application, and Data Architecture—detailing each domain's definition, purpose, value, and step‑by‑step guidance for creating enterprise architecture diagrams to align strategy, processes, and technology.
This article presents a detailed case study of how EasyPoint built a scalable, stable advertising platform using ClickHouse Enterprise, covering company background, data architecture with Kafka and Druid, ClickHouse advantages, serverless resource scaling, and extensive performance benchmarks.
This article explains the concept of query separation, outlines its suitable scenarios, compares implementation methods such as synchronous, asynchronous, and binlog approaches, discusses storage system choices like MongoDB, HBase, and Elasticsearch, and addresses consistency and operational challenges when decoupling read workloads from write workloads.
This article describes how 58.com designed and implemented a large‑scale user profiling platform using ClickHouse, covering system overview, core modules, major challenges of scale, complexity and performance, and the detailed storage, query, and optimization techniques applied to meet business needs.
This article provides a detailed overview of vector database structures, their evolution, enterprise challenges, functional features, future trends, and key use cases, illustrating how they serve as the memory engine for large AI models and support multimodal data processing in modern data architectures.