Tagged articles
Parquet
32 articles · Page 1 of 1
This article details integrating DuckDB into Spring Boot for embedded OLAP analytics, covering tenant-isolated connection pools, secure view-based file access, streaming exports via COPY and cursors, SQL injection guards, and performance tuning for ad-hoc Parquet/CSV queries without a separate cluster.
CSVDuckDBEmbedded Analytics
0 likes · 48 min read

This article explains how StarRocks queries Paimon Variant data, covering the differences between regular Parquet JSON and Variant, the three-phase read architecture, Shredding optimization for hot paths, practical SQL examples using get_variant_* functions, and a decision framework for choosing between JSON, Plain Variant, Shredded Variant, and formal typed columns.
PaimonParquetSQL
0 likes · 21 min read

This comprehensive DuckDB tutorial explains why loading large CSV files into Pandas is inefficient, demonstrates how DuckDB can query CSV, Parquet, and JSON directly, walks through SQL basics, advanced features like window functions and CTEs, compares performance against Pandas on ten‑million‑row datasets, and provides practical tips for tuning, common pitfalls, and when to choose DuckDB for data analysis.
DuckDBPandasParquet
0 likes · 28 min read

This article analytically examines INFINI Pizza, a Rust‑implemented distributed search database, detailing its design philosophy, hierarchical data model, rolling‑partition‑shard architecture, share‑nothing + io_uring I/O stack, true real‑time indexing, in‑place partial updates, AI‑native hybrid search capabilities, ecosystem components, and a point‑by‑point comparison with Elasticsearch.
AI NativeParquetReal-time
0 likes · 20 min read

This article explains why massive log systems must adopt hot‑cold separation, walks through the problem analysis, SLO definition, component design, Kafka partition planning, Elasticsearch and ClickHouse tuning, Parquet archiving, a unified query gateway with async cold queries, governance practices, cost modeling, common pitfalls, and a roadmap for evolving the platform.
ElasticsearchKafkaLog Storage
0 likes · 40 min read

The article explains how traditional Parquet‑based lakehouse architectures, optimized for large‑scale scans, struggle with AI workloads that need ultra‑low‑latency random access, and how Lance redesigns the storage format, indexing and write path to provide O(1) addressing, native vector support, and seamless integration with native execution engines.
AI workloadsLanceNative Execution
0 likes · 12 min read

This article breaks down Dremel's columnar encoding for nested data, detailing the definition‑level and repetition‑level concepts, showing step‑by‑step examples of encoding and reconstructing JSON‑like schemas, and explaining the limits of single‑column reconstruction.
Apache ArrowDremelParquet
0 likes · 9 min read

This article analyzes why StarRocks multi-table joins on Paimon/Iceberg tables remain significantly slower than internal tables even after cache warming, detailing six root causes across storage format, data layout, caching, execution engine, metadata, and version management, plus four optimization strategies to narrow the gap.
Join PerformanceOLAPPaimon
0 likes · 17 min read

The article explains how Apache Parquet’s columnar layout, multi‑level row‑group structure, projection and predicate push‑down, and advanced compression and encoding make it the high‑performance, space‑efficient storage format that powers modern big‑data ecosystems and tools like Spark, Python pandas, and ClickHouse.
ClickHouseCompressionParquet
0 likes · 11 min read

The article explains why CSV becomes a bottleneck for large‑scale data, demonstrates how Parquet’s columnar, typed, and compressed format dramatically reduces storage, speeds up reads, and improves data safety, and provides step‑by‑step Python code for migrating and benchmarking the switch.
CSVData EngineeringDuckDB
0 likes · 18 min read

This article presents seven practical DuckDB SQL patterns—querying files directly, treating partition folders as tables, deduplicating with QUALIFY, computing rolling metrics with window functions, pivot/unpivot, handling JSON arrays, and exporting results to Parquet—plus tips and a mini case study that show how to turn a notebook into a lightweight OLAP engine without leaving the Python environment.
DuckDBJSONParquet
0 likes · 12 min read

This article examines how adaptive structural encoding in columnar formats like Lance dramatically improves random‑access performance on NVMe storage, compares it with Apache Parquet and Arrow, and discusses the trade‑offs between scan speed, memory usage, and compression.
LanceNVMeParquet
0 likes · 17 min read

This article explains how Baidu’s Feed real‑time data warehouse was rebuilt using a pure streaming architecture, detailing the limitations of the previous stream‑batch design, the technical solutions—including core/non‑core data separation, metric calculation in streaming, and Parquet storage with Apache Arrow—and the resulting cost reductions, latency improvements, and future roadmap.
Apache ArrowParquetbatch processing
0 likes · 17 min read

This article explains how Baidu’s search data team redesigned its data warehouse with wide‑table modeling, Parquet columnar storage, and a Spark‑ClickHouse fusion engine, eliminating redundancy, cutting query latency from minutes to seconds, and enabling self‑service analytics for thousands of users.
Data ModelingETLParquet
0 likes · 21 min read

DuckDB's DuckLake is an open‑standard, SQL‑driven data lake and catalog format that simplifies lakehouse architecture by managing metadata in a database while storing data in scalable Parquet files, offering multi‑user collaboration, time‑travel queries, and MIT licensing.
DuckDBParquetdatabases
0 likes · 4 min read

The article introduces Archer, a new big‑data warehouse engine built on Iceberg that adds an inverted‑index mechanism using Tantivy to provide full‑text and JSON search, storage‑compute separation, and significant performance gains over traditional Elasticsearch and Iceberg connectors.
Archer EngineParquetTantivy
0 likes · 9 min read

ByteDance tackled massive feature‑storage challenges by replacing row‑based HDFS files with columnar Parquet and the Iceberg table format, enabling schema evolution, selective reads, efficient backfill, and training optimizations that cut storage costs by over 40% and reduced CPU and network I/O dramatically.
Parquetbig datadata lake
0 likes · 13 min read

The article proposes replacing the traditional multi‑layered data‑warehouse architecture (ODS‑DWD‑DWS‑ADS) with a single, column‑store wide‑table per business theme, achieving roughly 30 % storage savings and faster queries, while acknowledging higher ETL complexity, back‑tracking costs, and production timing challenges.
ETLParquetQuery Performance
0 likes · 11 min read

This article introduces the mongo2file library for converting MongoDB collections into various table formats such as CSV, Excel, JSON, pickle, feather, and parquet, explains its PyArrow dependency, shows installation and quick‑start code, discusses performance bottlenecks, and provides a full reference API.
ExcelMongoDBParquet
0 likes · 11 min read

This article provides a comprehensive introduction to Apache Parquet, covering its origins, columnar storage advantages, nested schema support, internal architecture, storage model components, comparison with ORC, and practical tools for inspecting Parquet files.
Data CompressionHadoopORC Comparison
0 likes · 10 min read

This article introduces Apache Iceberg as an open table format for massive analytic datasets, explains its underlying concepts such as schema, partitioning, statistics, and read/write APIs, compares it with Hive Metastore, outlines its ACID commit process, highlights the performance and operational advantages for big‑data workloads, and previews upcoming community features.
ACIDApache IcebergParquet
0 likes · 19 min read

Apache Beam 2.28.0 introduces extensive Parquet support, new hash functions in BeamSQL and ZetaSQL, ApproximateDistinct via HLL, enhanced I/O connectors including SpannerIO for Numeric fields, ParquetIO schema support, KafkaTableProvider thrift, HadoopFormatIO key/value cloning skip, and various other improvements.
Apache BeamI/O ConnectorsParquet
0 likes · 3 min read

This article details a real‑world investigation of Spark SQL job latency on a YARN cluster, explains how switching the scheduler to FAIR mode, creating resource pools, and consolidating small Parquet files dramatically reduced scheduler delay and cut execution time from over 100 seconds to under 20 seconds.
ParquetSparkSpark SQL
0 likes · 13 min read

This article provides a comprehensive overview of Apache Parquet, covering its modular architecture, nested data model, striping/assembly and definition level algorithms, file format details, push‑down optimizations, performance benchmarks, and the project's evolution within the big‑data ecosystem.
ParquetPushdown Optimizationcolumnar storage
0 likes · 18 min read

Lyft transformed its legacy data pipeline by designing a cloud‑native, Flink‑based near real‑time analytics platform that ingests billions of events, writes Parquet files to S3, leverages Presto for interactive queries, and implements multi‑stage non‑blocking ETL, fault‑tolerant back‑fill, and extensive performance optimizations.
AWSETLFlink
0 likes · 12 min read

This article examines how to run several Spark jobs concurrently on a shared SparkContext, balancing full CPU‑vcore utilization with the need to generate fewer Parquet files, and presents practical experiments, scheduling strategies, and performance results.
ParallelismParquetScala
0 likes · 12 min read

Apache Parquet is a language‑agnostic, columnar storage format for the Hadoop ecosystem that offers high compression, efficient I/O through column and predicate push‑down, nested‑structure support, and a three‑layer architecture, and is compared with ORC while providing tooling for schema inspection.
Apache HadoopData FormatsORC Comparison
0 likes · 9 min read

The article introduces Kyligence's Kylin on Parquet solution, explains its plug‑in architecture, reasons for replacing HBase with Parquet, details the new Spark‑based build and query engines, auto‑tuning, global dictionary, fault‑tolerance features, and presents performance comparisons with Kylin 3.0.
Apache KylinParquetSpark
0 likes · 11 min read


21CTONov 27, 2019 · Big Data
This article compares CSV, JSON, Parquet, and Avro file formats, outlining their structures, advantages, and drawbacks, and explains how Apache Spark supports each format for efficient big‑data storage and processing.
Apache SparkCSVJSON
0 likes · 8 min read

This article explains the evolution of Spark SQL from Shark, describes its key features such as SchemaRDD and in‑memory columnar storage, compares row‑based and column‑based storage, and provides practical Scala code examples for creating DataFrames and loading data from various sources.
JDBCParquetScala
0 likes · 16 min read

This article provides a comprehensive overview of Apache Parquet, covering its purpose, architectural components, nested data model, file structure, practical Hive commands for creating and inspecting Parquet tables, and a brief introduction to the TPC‑DS benchmark for performance testing.
Data ModelingHiveParquet
0 likes · 8 min read

This article reviews major big‑data storage and query architectures—including HBase, Dremel/Parquet, pre‑aggregation systems, Lucene, and the custom Tindex solution—evaluating their strengths, weaknesses, and suitability for real‑time, high‑volume analytical workloads.
HBaseLuceneParquet
0 likes · 20 min read
