Tagged articles
Apache Spark
61 articles · Page 1 of 1
This article compares JSONL (JSON Lines) with regular JSON, explaining how line-delimited records enable memory-efficient streaming for logs, machine learning datasets, and ETL pipelines, while detailing tool support in pandas, Spark, and Hugging Face, and clarifying when to stick with standard JSON.
Apache SparkETLHugging Face
0 likes · 21 min read

Apache Auron graduates from incubation to become a top-level project after 13 months, delivering 2x faster Spark query performance and 50% resource savings via its Rust-native vectorized execution engine, already proven at exabyte scale across companies like Kuaishou, DiDi, and LinkedIn.
Apache ArrowApache AuronApache Spark
0 likes · 14 min read

Apache Spark 4.2 introduces a lightweight Spark Connect architecture, native AI integration, enhanced Metrics View for unified semantics, Arrow‑first performance gains, advanced SQL extensions like vector search and QUALIFY, robust geospatial support, and a revamped streaming engine with auto CDC and sub‑millisecond state cleanup.
Apache SparkArrowCDC
0 likes · 13 min read

Apache Spark 4.0 introduces a high‑performance VARIANT data type for semi‑structured JSON, native SQL UDFs that eliminate Python UDF bottlenecks, a richer Python DataSource API, a new pipeline syntax, upgraded Structured Streaming state management, and Alibaba Cloud EMR Serverless optimizations that together deliver up to 30% speed gains and seamless migration from Spark 3.x.
Apache SparkPython APISQL UDF
0 likes · 12 min read

Spark 4.0 delivers 20‑50% performance gains and new features like Spark Connect, VARIANT types, and enhanced SQL, but it also introduces breaking changes such as mandatory JDK 17, dropping Scala 2.12, default ANSI mode, removal of Mesos, and altered JDBC type mappings, requiring careful planning and staged migration to avoid runtime failures.
ANSI modeApache SparkJDK 17
0 likes · 19 min read

The article details how China Postal Savings Bank rebuilt its core account transaction query system with a distributed micro‑service and big‑data stack, achieving millisecond‑level responses for over ten years of data, boosting performance by 700% and enabling new intelligent services.
Apache SparkFinancial ComplianceGaussDB
0 likes · 9 min read

The 2025 release of Apache Spark 4.0 brings a comprehensive overhaul—including default ANSI SQL mode, full SQL scripting support, a new Real‑Time streaming mode, adaptive query execution, dynamic memory management, and GPU‑accelerated MLlib—significantly boosting performance, reliability, and developer productivity across big‑data workloads.
Apache SparkGPU accelerationSQL
0 likes · 9 min read

This guide outlines a step‑by‑step learning path for Apache Spark, covering core concepts, environment setup, hands‑on WordCount code, API mastery, ecosystem extensions like Structured Streaming and MLlib, deployment options, performance tuning, and practical project advice.
Apache SparkPySparkdata processing
0 likes · 7 min read

Uber migrated more than two million daily Spark applications from version 2.4 to 3.3, detailing the motivations, architecture, four-step migration process, custom tools like Polyglot Piranha and Iron Dome, and the resulting performance, cost, and productivity gains.
Apache SparkData EngineeringIron Dome
0 likes · 11 min read

This article details how SF Technology adopted Apache Gluten with Velox to accelerate Spark queries, describing the architecture, task lifecycle, management framework, simulation system, unified SQL, fallback mechanisms, dynamic memory tuning, columnar shuffle, and future plans that together saved over 10,000 CPU cores and reduced operator fallback rates to around 4%.
Apache SparkGlutenVelox
0 likes · 16 min read

This article introduces PySpark as the Python API for Apache Spark, explains Spark's core concepts and advantages, details PySpark's main components and a simple code example, compares it with Pandas, and outlines typical big‑data scenarios and further learning directions.
Apache SparkDataFramesPySpark
0 likes · 5 min read

This article explains Spark SQL's window function fundamentals, introduces two key optimizations—Offset Window Frame and Infer Window Group Limit—and provides a detailed Q&A covering implementation details, execution plan impacts, and underlying architecture.
Apache SparkSQL PerformanceSpark SQL
0 likes · 13 min read

This article explains how Adaptive Query Execution (AQE) in Apache Spark 4.0 dynamically optimizes query plans through features such as join reordering, partition pruning, skew handling and coalescing, delivering significant performance gains, resource efficiency and reduced manual tuning across real‑world big‑data workloads.
Apache SparkSpark 4.0adaptive query execution
0 likes · 13 min read

Baidu’s DATAPILOT platform combines natural‑language interaction with GPU‑accelerated Spark‑RAPIDS to turn complex, multi‑table SQL queries into seconds‑fast results, boosting ad‑revenue analysis efficiency by up to five‑fold while reducing infrastructure costs.
Apache SparkBaiduGPU acceleration
0 likes · 10 min read

This article shares Xiaomi's experience building a next‑generation one‑stop data development platform on Spark 3.1, covering typical challenges such as Multiple Catalog implementation, Hive‑SQL to Spark‑SQL migration, offline Spark performance and stability optimizations, and future roadmap plans.
Apache SparkSQL migrationbig data
0 likes · 18 min read

This article provides a comprehensive overview of Apache Spark SQL, covering its fundamental concepts such as TreeNode, AST, and QueryPlan, the distinction between logical and physical plans, the rule‑execution framework, core components like SparkSqlParser and Analyzer, as well as the Spark Session, Dataset/DataFrame, and various writer APIs, supplemented by a detailed Q&A session.
Apache SparkSQL OptimizationSpark SQL
0 likes · 19 min read

This article provides a comprehensive overview of Apache Spark, covering its origins, key characteristics, core concepts such as RDD, DAG, partitioning and dependencies, the internal architecture including SparkConf, SparkContext, SparkEnv, storage and scheduling systems, as well as deployment models and the company behind the product.
Apache SparkRDDSpark Architecture
0 likes · 16 min read

Meituan enhances Apache Spark by integrating the Gluten‑Velox vectorized execution engine, converting row‑wise operations to columnar SIMD processing, which yields over 40 % memory savings and up to 13 % faster runtimes across thousands of ETL jobs, while addressing stability, ORC support, shuffle redesign, and off‑heap memory optimization.
Apache SparkC++Gluten
0 likes · 30 min read

This guide introduces Apache Spark's Python API, showing how to install PySpark, launch an interactive shell, create a SparkSession, read and write data from various sources, perform transformations, and apply key performance‑tuning practices for efficient big‑data processing.
Apache SparkPerformance TuningPySpark
0 likes · 5 min read

This guide walks you through using Alibaba Cloud EMR Serverless Spark to ingest OSS‑HDFS audit logs, create source, detail, and summary data‑warehouse tables with Spark SQL, orchestrate the tasks in a workflow, and schedule daily IP‑traffic analysis, complete with code snippets and UI screenshots.
Apache SparkEMR Serverless SparkSQL
0 likes · 12 min read

This article shares Xiaomi's real‑world challenges and solutions when building a new Spark 3.1‑based data platform, covering Multiple Catalog implementation, Hive‑to‑Spark SQL migration, automated batch upgrades, performance and stability optimizations, and future roadmap for vectorized execution.
Apache SparkHiveMetacat
0 likes · 14 min read

Airbnb’s Riverbed framework unifies streaming CDC events and batch Spark jobs behind a GraphQL‑based declarative API to automatically build and maintain distributed materialized views, using Kafka‑partitioned ordering and version control to deliver billions of daily updates with low‑latency reads for features such as payments and search.
AirbnbApache SparkData Engineering
0 likes · 8 min read

Apache Celeborn (Incubating) is a remote shuffle service designed to overcome the inefficiencies, high storage demands, network overhead, and limited fault tolerance of traditional Spark shuffle implementations by introducing push‑shuffle, partition splitting, columnar shuffle, multi‑layer storage, and elastic, stable, and scalable architectures.
Apache SparkRemote ShuffleShuffle Service
0 likes · 15 min read

This article compares traditional data warehouses, modern data lakes, and emerging lakehouse architectures, explaining their design patterns, advantages, disadvantages, and suitable use cases, while detailing implementation considerations such as schema design, ETL/ELT processes, file formats like Delta, Iceberg, and Hudi, and factors influencing platform selection.
Apache SparkDelta LakeETL
0 likes · 20 min read

iQIYI now relies on Apache Spark as its main offline engine, processing over 200 000 daily tasks for ETL, data synchronization and analytics, while recent optimizations—dynamic resource allocation, adaptive query execution, compression, rebalance, Z‑order and resource‑governance—have cut compute usage by ~27 %, storage by up to 76 % and improved query speed, completing a large‑scale migration from Hive and paving the way for Spark 3.4 and Iceberg support.
Apache SparkSQL Servicedata lake
0 likes · 21 min read

This article reviews the limitations of traditional Spark shuffle, introduces Apache Celeborn (Incubating) as a remote shuffle service, and details its design for performance, stability, and elasticity, including push shuffle, partition splitting, columnar shuffle, multi‑layer storage, congestion control, and real‑world evaluation.
Apache SparkShuffle Servicebig data
0 likes · 19 min read

This article details how NetEase Yanxuan transformed its data platform from a dual Lambda architecture to a unified batch‑stream solution built on Apache Iceberg, covering the original challenges, the evaluation of Iceberg versus Hudi and Delta Lake, implementation of stream‑batch pipelines, message ordering fixes, snapshot generation, and extensive table‑governance optimizations.
Apache FlinkApache SparkBatch-Stream Integration
0 likes · 14 min read

This presentation details how Guanyuan Data leverages Apache Spark within its self‑service analytics platform, covering product features, flexible deployment, resource isolation, performance challenges, architectural solutions, and future cloud‑native enhancements to support thousands of users and massive query workloads.
Apache SparkSelf-Service Analyticsbig data
0 likes · 14 min read

This article explains the growing importance of data lineage in large data warehouses, evaluates three Spark lineage extraction approaches, and provides a detailed, step‑by‑step guide to integrating the open‑source Spline agent—including codeless and programmatic initialization, configuration, dispatcher setup, post‑processing, and known limitations.
Apache SparkSparkSpline
0 likes · 16 min read

This article presents a detailed overview of Apache Spark's Adaptive Query Execution evolution, its optimization techniques, and performance gains, followed by an in‑depth discussion of Apache Kyuubi's architecture, security integrations, cloud‑native capabilities, and practical Rebalance + Z‑Order strategies that enhance data‑warehouse task efficiency and query performance.
Apache SparkBig Data OptimizationKyuubi
0 likes · 19 min read

This article introduces Apache SeaTunnel, a distributed, high‑performance data integration platform built on Spark and Flink, outlines its technical features, workflow, and plugin ecosystem, and details a concrete traffic‑management use case involving incremental Oracle‑to‑warehouse data synchronization with Spark resources and scheduled shell scripts.
Apache FlinkApache SparkETL
0 likes · 12 min read

This article introduces Apache Kyuubi—a multi‑tenant Thrift JDBC/ODBC service built on Spark—detailing its architecture, advantages over Spark Thrift Server, real‑world use cases, open‑source community progress, and practical deployment strategies on mobile cloud, Kubernetes, and with Trino.
Apache SparkKubernetesKyuubi
0 likes · 16 min read

EMR Studio is a one‑stop, open‑source‑compatible big data development platform that integrates Zeppelin, Jupyter, Airflow and a custom Cluster Manager to streamline job creation, scheduling, monitoring, and cluster switching, thereby addressing common usability challenges in Spark, Flink, Hive, and Presto workflows.
AirflowApache SparkData Engineering
0 likes · 9 min read

This article provides a thorough learning roadmap for Apache Spark, covering its background papers, core concepts such as RDD and fault tolerance, module breakdown, recommended books and repositories, source‑code reading tips, hands‑on projects, and interview‑oriented optimization guidance.
Apache SparkRDDSpark SQL
0 likes · 15 min read

This article explains how to create a custom SparkListener in Apache Spark, provides Scala code examples for the listener and a main application, and details two registration approaches—via Spark configuration or SparkContext—along with a comprehensive list of listener event methods.
Apache SparkScalaSpark
0 likes · 5 min read

This article explains how NVIDIA's RAPIDS Accelerator leverages GPUs to speed up Apache Spark 3.0 workloads, detailing the underlying architecture, benchmark results on TPC‑DS and recommendation models, required configuration changes, supported operators, shuffle optimizations, and the enhancements introduced in versions 0.2 and 0.3.
Apache SparkGPU accelerationRAPIDS
0 likes · 19 min read

Apache Spark Core is a high‑performance, fault‑tolerant engine that abstracts distributed computation through SparkContext, DAG and Task schedulers, supports in‑memory and disk storage, runs on various cluster managers (YARN, Kubernetes, etc.), and unifies batch, streaming, ML and graph processing via its rich ecosystem.
Apache SparkDAG schedulerRDD
0 likes · 17 min read

Delta Lake adds ACID transaction support, schema enforcement, data versioning, and unified batch‑and‑stream processing to Apache Spark‑based data lakes, addressing reliability, quality, performance, and update challenges of traditional data lake architectures.
ACID transactionsApache SparkDelta Lake
0 likes · 13 min read

Apache Spark 3.0 introduces a suite of performance enhancements, richer APIs, improved monitoring, SQL compatibility, new data sources, and ecosystem extensions, including Adaptive Query Execution, Dynamic Partition Pruning, Join Hints, pandas UDF improvements, and accelerator‑aware scheduling, to boost scalability and ease of use for big‑data workloads.
Apache SparkData EngineeringSpark 3.0
0 likes · 15 min read

Apache Spark 3.0 introduces extensive SQL performance enhancements, including a new explain format, expanded join hints, adaptive query execution, dynamic partition pruning, enhanced nested column pruning, improved aggregation code generation, and support for newer Scala and Java versions, all aimed at optimizing query planning and execution.
Apache SparkPerformance TuningSQL Optimization
0 likes · 14 min read

Apache Spark 3.0.0, released after a 21‑month development cycle and several preview and release‑candidate votes, introduces major enhancements such as Dynamic Partition Pruning, Adaptive Query Execution, accelerator‑aware scheduling, DataSource V2, expanded pandas UDFs, new join hints, richer monitoring, SparkR vectorization, Kafka header support, and broader ecosystem integrations, while fixing over 3,400 issues.
Apache SparkDataSource V2Dynamic Partition Pruning
0 likes · 17 min read

Apache Spark 3.0, released after a 21‑month development cycle, introduces dynamic partition pruning, adaptive query execution, accelerator‑aware scheduling, DataSource V2, enhanced pandas UDFs, new join hints, richer monitoring, ANSI‑SQL compatibility, SparkR vectorization, Kafka header support, and numerous platform upgrades, all backed by over 3,400 resolved issues.
Apache SparkDataSource V2Dynamic Partition Pruning
0 likes · 17 min read

This article explains why Spark is a mature big‑data framework, recommends which Spark versions to study, lists essential research papers, describes how to set up the development environment, and outlines the key components of Spark’s core architecture for effective source‑code exploration.
Apache SparkRDDbig data
0 likes · 6 min read


21CTONov 27, 2019 · Big Data
This article compares CSV, JSON, Parquet, and Avro file formats, outlining their structures, advantages, and drawbacks, and explains how Apache Spark supports each format for efficient big‑data storage and processing.
Apache SparkCSVJSON
0 likes · 8 min read

This article explains the origins and motivations of Delta Lake, details its ACID transaction support, schema enforcement, metadata handling, versioning, and unified batch‑and‑stream processing, and provides a step‑by‑step Maven and Spark code tutorial for creating, updating, and querying Delta tables.
ACIDApache SparkDelta Lake
0 likes · 10 min read

The article chronicles JD engineer Zheng Ruifeng’s path to becoming a Spark Committer, highlighting his early involvement, key contributions to Spark’s ML and GraphX components, the community’s scale, and his vision for future improvements in the big‑data platform.
Apache SparkCommitterJD.com
0 likes · 6 min read

Despite recent market turbulence and negative headlines, Hadoop's revenue continues to grow, driven by cloud migration, evolving storage solutions, and increasing adoption of related projects like Spark and Kafka, positioning it as a leading data‑lake technology.
Apache SparkHadoopObject Storage
0 likes · 8 min read

This article explains Apache Spark's memory architecture, covering the shift from static to unified memory management, the roles of on‑heap and off‑heap memory, configurable parameters, dynamic memory sharing between execution and storage, and the legacy mode introduced in Spark 1.6.
Apache SparkExecutorOff‑Heap
0 likes · 7 min read

The article provides a comprehensive overview of recent Apache Spark advancements—including Delta Lake, Data Source V2, runtime optimizations, relational cache, cloud‑native challenges, AI integration via Project Hydrogen, and the anticipated features of Spark 3.0—highlighting how these innovations address modern data‑warehouse, cloud, and machine‑learning workloads.
Apache SparkDelta LakeRuntime Optimization
0 likes · 17 min read

Apache Spark, an open‑source unified analytics engine from UC Berkeley’s AMP Lab, is the leading platform for large‑scale batch and streaming data processing, featuring components such as Spark SQL, Streaming, GraphX, MLlib, and core modules like DAGScheduler, TaskScheduler and BlockManager.
Apache SparkBlockManagerDAGScheduler
0 likes · 4 min read

This article continues the deep dive into Apache Spark memory management, explaining storage memory handling—including RDD persistence, caching, eviction, and disk spilling—as well as execution memory allocation for multi-tasking and shuffle operations, and detailing Spark’s internal structures such as BlockManager, StorageLevel, and Tungsten page management.
Apache SparkRDD PersistenceTungsten
0 likes · 13 min read

This article compares Apache Spark and Apache Flink, outlining their programming models, streaming mechanisms, state management, time semantics, and exactly‑once guarantees, and highlights the strengths and differences of each framework for batch and real‑time big‑data processing.
Apache FlinkApache SparkState Management
0 likes · 8 min read

This article explains the definition of big data, its characteristic 3Vs, common data sources, supporting IT infrastructure, key technologies such as Hadoop and Spark, specialized databases, required skills, and several practical business use cases.
Apache SparkHadoopNoSQL
0 likes · 8 min read

Apache Spark 2.3 introduces major upgrades such as millisecond‑latency continuous streaming, stream‑to‑stream joins, a native Kubernetes scheduler backend, accelerated Pandas UDFs, and several MLlib improvements, all aimed at making big‑data processing faster, easier, and smarter.
Apache SparkContinuous ProcessingMLlib
0 likes · 7 min read

This article details how Suning built a Hadoop‑based big data platform and leveraged Apache Spark to process terabytes of product price and inventory data, describing the system architecture, four key technical practices, performance results, and future data‑lake directions.
Apache SparkDataFramesETL
0 likes · 12 min read

This article provides a comprehensive overview of Apache Spark's architecture, covering its RDD abstraction, computation model, various cluster deployment modes, RPC communication layer, startup procedures, core components, interaction flows, and block management for broadcast variables.
Apache SparkCluster ModeRDD
0 likes · 15 min read

This article shares hands‑on experience from Spark Summit attendees, covering why Spark is powerful, common performance problems such as slow jobs, OOM, data skew, excessive partitions, and provides concrete tuning advice on executors, cores, memory, and debugging techniques.
Apache SparkData SkewExecutor Configuration
0 likes · 11 min read

The article explains how Spark 2.0’s second‑generation Tungsten engine replaces the traditional Volcano iterator model with whole‑stage code generation and vectorization, eliminating virtual calls, keeping temporary data in CPU registers, and using loop unrolling and SIMD to achieve order‑of‑magnitude performance gains on large‑scale data workloads.
Apache SparkTungstenWhole-stage code generation
0 likes · 12 min read


ITPUBJul 10, 2016 · Big Data
This article examines Apache Spark's interactive mode performance, revealing that while small datasets respond within seconds, processing beyond about 1 TB dramatically increases latency, and it discusses practical limits, hardware considerations, and the need to preload large datasets from disk.
Apache SparkResponse Timebig data
0 likes · 5 min read


21CTOMar 30, 2016 · Big Data
This article explains Apache Spark’s core concepts, the RDD programming model, how Spark runs on YARN with driver and executor nodes, the distinction between transformations and actions, partitioning strategies, and an overview of SparkSQL processing.
Apache SparkRDDSparkSQL
0 likes · 18 min read

The article compares Apache Storm and Apache Spark, examining their origins, architecture, language support, integration capabilities, and performance characteristics, and offers guidance on selecting the right platform for real‑time business intelligence based on specific workload and infrastructure needs.
Apache SparkApache Stormbig data
0 likes · 11 min read
