Data Systems | BestHub

DataFunTalk

Jul 17, 2024 · Databases

From DIKW to Distributed Data Warebase: Letting Data Emerge as Intelligence

This article explores the DIKW hierarchy, explains how data evolves into information, knowledge, and wisdom, examines traditional data models and products, critiques existing multi‑system architectures, and proposes a new distributed Data Warebase that unifies structured, semi‑structured, and vectorized knowledge to enable intelligent data-driven applications.

Artificial IntelligenceDIKWData Systems

0 likes · 24 min read

From DIKW to Distributed Data Warebase: Letting Data Emerge as Intelligence

DataFunSummit

Jun 21, 2024 · Big Data

Building a Complete Data System with Apache Arrow: Architecture, Dynamic Schema Modeling, and Practical Tips

This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes dynamic read‑time modeling, outlines the system’s execution flow, storage and indexing strategies, and shares practical tips and extensions for building scalable big‑data solutions.

AceroApache ArrowBig Data

0 likes · 20 min read

Building a Complete Data System with Apache Arrow: Architecture, Dynamic Schema Modeling, and Practical Tips

DataFunSummit

Apr 23, 2024 · Big Data

Building a Data System with Apache Arrow: Design, Implementation, and Practical Tips

This article explains why new data systems are needed, introduces Apache Arrow’s columnar in‑memory format and its zero‑copy advantages, describes how to model data at read time, outlines the execution flow with Acero and SQL planning, and shares practical tips and extensions for building robust, dynamic‑schema data platforms.

AceroApache ArrowBig Data

0 likes · 20 min read

Building a Data System with Apache Arrow: Design, Implementation, and Practical Tips

Sohu Tech Products

Mar 6, 2024 · Big Data

Building Data Systems with Apache Arrow: Architecture, Memory Format, and Execution

The article explains how Apache Arrow’s columnar, cross‑language in‑memory format enables high‑performance, interoperable data systems—replacing traditional row‑oriented databases—by supporting dynamic schemas, zero‑copy data exchange, efficient indexing, Acero‑based query execution, and Flight/ADBC connectivity, while offering practical guidance and highlighting challenges.

Apache ArrowBig DataData Systems

0 likes · 20 min read

Building Data Systems with Apache Arrow: Architecture, Memory Format, and Execution

DataFunTalk

Feb 28, 2024 · Big Data

Building a Data System with Apache Arrow: Design, Modeling, and Execution

This article explains why new data systems are needed, introduces Apache Arrow and its columnar in‑memory format, describes read‑time modeling and dynamic schema handling, and shows how Arrow can be used to build a complete data processing pipeline with indexing, SQL planning, and zero‑copy data exchange.

Apache ArrowBig DataData Systems

0 likes · 20 min read

Building a Data System with Apache Arrow: Design, Modeling, and Execution

DevOps

Dec 14, 2023 · Operations

Data‑Centric Reflections on DevOps Efficiency and System Design

This article examines how viewing software development through a data‑system lens can enrich DevOps practices, discussing value streams, microservice decomposition, agile testing techniques, implementation principles, and the role of continuous delivery in reducing uncertainty and enhancing organizational effectiveness.

Continuous DeliveryData SystemsDevOps

0 likes · 10 min read

Data‑Centric Reflections on DevOps Efficiency and System Design

DataFunSummit

Oct 24, 2023 · Big Data

Using Apache Arrow to Quickly Build Modern Data Systems

This announcement introduces Li Chenxi, a big‑data R&D engineer, and outlines his talk on leveraging Apache Arrow’s columnar in‑memory format to efficiently construct modern, read‑time modeling data systems, highlighting key features, ecosystem, and practical implementation benefits for the audience.

Apache ArrowBig DataColumnar Memory

0 likes · 2 min read