Tagged articles
Spark SQL
39 articles · Page 1 of 1
A leading automaker processes hundreds of millions of road images using Alibaba Cloud EMR Serverless Spark AI Functions for scenario extraction and vector embedding, integrating multimodal model calls into Spark jobs via SQL, PySpark, and Ray to enable efficient retrieval and sample selection for autonomous driving training.
AI FunctionEMR Serverless SparkSpark SQL
0 likes · 12 min read

This article provides a comprehensive, step‑by‑step guide to mastering Spark SQL in high‑concurrency, billion‑row scenarios, covering the execution chain, architecture layers, Delta Lake integration, performance tuning, production‑grade streaming and batch pipelines, Kubernetes deployment, parameter management, observability, and real‑world case studies.
Data EngineeringDelta LakeKubernetes
0 likes · 44 min read

This comprehensive guide walks you through Spark SQL fundamentals—including DataFrame and Dataset APIs—delves into the Catalyst optimizer and Tungsten engine, presents practical Java examples, and shares concrete tuning techniques and real-world ETL scenarios for handling large‑scale data.
CatalystDatasetETL
0 likes · 8 min read

This article explains how to reduce intermediate tables, simplify dependencies, and improve execution efficiency in Spark SQL pipelines by using session‑level temporary views and regex‑based SQL parsing to automatically merge and rewrite DAG tasks in large‑scale data platforms.
DAG OptimizationETLSpark SQL
0 likes · 13 min read

This article explains Spark SQL’s Catalyst optimizer, describing its extensible design, tree‑based representation, rule‑driven transformations, batch execution to a fixed point, and how Scala’s pattern matching and quasiquotes enable efficient analysis, logical optimization, physical planning, and code generation.
Catalyst OptimizerScalaSpark SQL
0 likes · 18 min read

This article details the evolution of a fast‑growing finance reporting system from a monolithic microservice architecture plagued by data inconsistency, low efficiency, and scalability limits to a robust, high‑performance big‑data warehouse built with layered data models, SparkSQL processing, and unified scheduling, highlighting design decisions, technical trade‑offs, and measurable performance gains.
Spark SQLarchitecture evolutiondata warehouse
0 likes · 23 min read

This article details the evolution of a fast‑growing e‑commerce finance platform from a monolithic microservice architecture plagued by data inconsistency, low processing efficiency, and scalability limits to a robust, distributed big‑data warehouse using SparkSQL, layered data models, and optimized scheduling, achieving ten‑fold performance gains and near‑zero failure rates.
ETLSpark SQLarchitecture
0 likes · 21 min read

This article explains the underlying principles of the Spark SQL optimizer and presents three classic optimization paradigms—push‑down optimization, operator elimination/merging, and expression elimination/replacement—illustrating each with concrete rule implementations and code examples.
OptimizerSpark SQLbig data
0 likes · 12 min read

This article explains Spark SQL's window function fundamentals, introduces two key optimizations—Offset Window Frame and Infer Window Group Limit—and provides a detailed Q&A covering implementation details, execution plan impacts, and underlying architecture.
Apache SparkSQL PerformanceSpark SQL
0 likes · 13 min read

This article examines Spark SQL expression-level optimizations, focusing on redesigning LIKE ALL and LIKE ANY to reduce memory and stack usage, refactoring the TRIM function for better code reuse and performance, and implementing constant folding to cache computed constant expressions, thereby enhancing query efficiency in big-data workloads.
Expression OptimizationPerformance TuningSpark SQL
0 likes · 16 min read

This talk, the third in a Spark series, introduces the Spark SQL parsing layer, explains its architecture and integration with ANTLR4, details core implementation classes, and presents a real‑world optimization case that reduces code complexity and improves maintainability.
Antlr4ScalaSpark SQL
0 likes · 15 min read

This article provides a comprehensive overview of Apache Spark SQL, covering its fundamental concepts such as TreeNode, AST, and QueryPlan, the distinction between logical and physical plans, the rule‑execution framework, core components like SparkSqlParser and Analyzer, as well as the Spark Session, Dataset/DataFrame, and various writer APIs, supplemented by a detailed Q&A session.
Apache SparkSQL OptimizationSpark SQL
0 likes · 19 min read

This article shares Xiaomi's real‑world challenges and solutions when building a new Spark 3.1‑based data platform, covering Multiple Catalog implementation, Hive‑to‑Spark SQL migration, automated batch upgrades, performance and stability optimizations, and future roadmap for vectorized execution.
Apache SparkHiveMetacat
0 likes · 14 min read

This guide explains how to perform incremental queries on Hudi tables by configuring Hive synchronization, using Spark SQL both programmatically and via pure SQL, and leveraging Flink SQL in batch and streaming modes, with detailed parameter settings and code examples.
Flink SQLHiveHudi
0 likes · 20 min read

This article details iQIYI's implementation of Apache Kyuubi for Spark Thrift Server, covering the evolution from native Spark Thrift to Kyuubi 0.7 and 1.x, multi‑tenant architecture, tag‑based configurations, SQL auditing, lineage collection, service monitoring, small‑file and Z‑order optimizations, and a brief Q&A.
Apache KyuubiSpark SQLdata platform
0 likes · 15 min read

iQIYI accelerated its big‑data platform by migrating the OLAP layer from Hive to Spark SQL, achieving a 67 % speedup, 50 % CPU reduction and 44 % memory savings, while automating the conversion of tens of thousands of tasks and delivering faster analytics for advertising, BI, membership and user‑growth services.
HiveSpark SQLdata migration
0 likes · 18 min read

This article details how Ping An Insurance migrated its offline Hive SQL workloads to Spark SQL, covering business background, deployment mode selection, migration workflow, typical challenges, optimization measures, and the resulting performance and resource utilization improvements.
Cluster MigrationDeployment ModesSpark SQL
0 likes · 16 min read

The article provides an in‑depth overview of Apache Hudi 0.11.0, covering its new multi‑level index design, Spark SQL enhancements, Flink integration improvements, and additional performance and usability features aimed at boosting read/write efficiency in large‑scale data lake environments.
Apache HudiFlink IntegrationMulti-Level Index
0 likes · 15 min read

This article details the Snowball data team's migration from Hive3 on Tez to Spark SQL, covering the motivations, comparative performance tests, encountered compatibility issues, configuration work‑arounds, and future plans for consolidating ETL workloads on Spark.
ETLHiveSpark SQL
0 likes · 13 min read

The article details how to add field‑lineage tracking to Spark SQL by creating a custom SparkSessionExtension that injects a check‑analysis rule and a parser, which capture INSERT statements, analyze the physical plan, and generate a JSON mapping of source‑to‑target fields for data governance.
Logical PlanPhysical PlanSQL Extensions
0 likes · 9 min read

This article provides a step‑by‑step guide on integrating Apache Hudi with Hive and Presto, demonstrates core Hudi operations such as insert, upsert, delete, query, and Hive synchronization using Scala code, and shows how to manage Hudi tables through Spark SQL DDL/DML commands.
Apache HudiHiveSpark SQL
0 likes · 16 min read

This article provides an in‑depth overview of Spark SQL, covering its architecture, DataSet/DataFrame creation, DSL and SQL usage, integration with Hive, custom UDF/UDAF/Aggregator implementations, handling of small files, Cartesian product detection, and a catalog of useful built‑in functions and window operations.
DatasetHiveSpark SQL
0 likes · 29 min read

This article provides a comprehensive overview of Spark SQL, covering its architecture, DataSet/DataFrame APIs, code examples for creating and querying datasets, join strategy selection, handling Hive tables, small‑file issues, inefficient NOT‑IN subqueries, Cartesian products, and a catalog of useful built‑in functions.
DatasetHive IntegrationSpark SQL
0 likes · 40 min read

This article provides a thorough learning roadmap for Apache Spark, covering its background papers, core concepts such as RDD and fault tolerance, module breakdown, recommended books and repositories, source‑code reading tips, hands‑on projects, and interview‑oriented optimization guidance.
Apache SparkRDDSpark SQL
0 likes · 15 min read

This tutorial demonstrates how to use Spark SQL to create, insert, update, delete, merge, and drop Apache Hudi tables, covering environment setup, Spark‑SQL launch, configuration, and a series of SQL commands with example outputs.
Apache HudiSQLSpark SQL
0 likes · 7 min read

This article explains the small‑file problem in Spark SQL on HDFS, its impact on NameNode memory and query performance, describes how dynamic partition inserts and shuffle settings generate many files, and presents practical solutions such as partition‑based distribution, random bucketing and adaptive query execution to control file count.
HadoopSpark SQLadaptive query
0 likes · 12 min read

This article details a real‑world investigation of Spark SQL job latency on a YARN cluster, explains how switching the scheduler to FAIR mode, creating resource pools, and consolidating small Parquet files dramatically reduced scheduler delay and cut execution time from over 100 seconds to under 20 seconds.
ParquetSparkSpark SQL
0 likes · 13 min read

This article explains how Tubi customizes Spark SQL using lightweight macro‑based extensions to simplify column exclusion, JSON path queries, and other complex operations without modifying Spark's source code, detailing the two‑stage processing, example macros, and benefits for big‑data workloads.
Custom SQLMacrosScala
0 likes · 9 min read

This article provides a comprehensive guide on integrating Hive with Spark SQL, covering Hive‑on‑Spark and Spark‑on‑Hive setups, spark‑shell and spark‑sql usage, HiveServer2 with Beeline, Scala scripts for reading and writing Hive tables, and partition handling for aggregated results.
HiveHiveServer2Scala
0 likes · 7 min read

This article compares Spark SQL and Presto, explaining their architectures, key differences, performance characteristics, supported connectors, installation requirements, and typical use cases, while providing head‑to‑head tables and examples of federated queries.
SQL EnginesSpark SQLcomparison
0 likes · 10 min read

This article explains how Spark SQL processes joins—including inner, outer, semi, and anti joins—by describing the overall query planning flow, the three physical join strategies (sort‑merge, broadcast, and hash), and the specific implementation details for each join type.
DataFramesSQL OptimizationSpark SQL
0 likes · 10 min read

The article explains Spark SQL's Catalyst optimizer rules for selecting among Broadcast hash join, Shuffle hash join, and Sort‑merge join, covering build‑side determination, size thresholds, broadcast hints, local hash‑map construction, and fallback strategies for non‑equi joins.
Broadcast JoinShuffle Hash JoinSort Merge Join
0 likes · 10 min read

ByteDance’s data warehouse team shares comprehensive optimizations for Spark SQL, covering architecture overview, bucket join enhancements, materialized columns and views, and shuffle stability and performance improvements, illustrating practical techniques that boost query efficiency and job reliability in large‑scale big‑data environments.
Materialized ColumnsShuffle OptimizationSpark SQL
0 likes · 20 min read

Apache Spark, an open‑source unified analytics engine from UC Berkeley’s AMP Lab, is the leading platform for large‑scale batch and streaming data processing, featuring components such as Spark SQL, Streaming, GraphX, MLlib, and core modules like DAGScheduler, TaskScheduler and BlockManager.
Apache SparkBlockManagerDAGScheduler
0 likes · 4 min read

This article provides a comprehensive overview of Apache Parquet, covering its purpose, architectural components, nested data model, file structure, practical Hive commands for creating and inspecting Parquet tables, and a brief introduction to the TPC‑DS benchmark for performance testing.
Data ModelingHiveParquet
0 likes · 8 min read

This article presents the integration of Alluxio as an in‑memory caching layer for QueryEngine's Spark SQL engine, detailing Alluxio's architecture, key features, deployment practice, performance testing methodology, results, and lessons learned for large‑scale ad‑hoc query acceleration.
AlluxioSpark SQLStorage
0 likes · 13 min read

This article shares a comprehensive set of practical tips and solutions for common Spark SQL issues—including out‑of‑memory errors, UDF‑induced GC, thread blocking, system‑property initialization, speculation side‑effects, accumulator traps, concurrent job scheduling, and excessive logging—helping engineers improve stability and efficiency of their Spark‑based financial systems.
AccumulatorPerformance TuningSpark
0 likes · 15 min read


ITPUBMar 22, 2017 · Big Data
This article walks through Spark’s origins, its core RDD concept, how it improves on Hadoop’s MapReduce, the role of in‑memory processing, functional programming support, and the emergence of Spark SQL with DataFrames and the Catalyst optimizer.
MapReduceRDDSpark
0 likes · 25 min read

This article provides a comprehensive guide on tackling Spark performance bottlenecks by diagnosing data skew, locating the offending stages and operators, and applying a range of practical solutions—including Hive pre‑processing, key filtering, shuffle parallelism, two‑stage aggregation, map‑join, and combined strategies—followed by an in‑depth discussion of shuffle manager evolution and key configuration parameters for fine‑tuning.
Data SkewPerformance TuningShuffle Optimization
0 likes · 35 min read
