0 views collected around this technical thread.
This article introduces PySpark as the Python API for Apache Spark, explains Spark's core concepts and advantages, details PySpark's main components and a simple code example, compares it with Pandas, and outlines typical big‑data scenarios and further learning directions.
This article teaches how to use Pandas for data reshaping, covering the melt method for converting wide-format data to long-format and the pivot method for the reverse transformation, with practical code examples.
This article explains the differences between pandas loc/iloc and at/iat, demonstrates why using loc/iloc inside loops can be extremely slow, and shows how replacing them with at/iat can speed up DataFrame updates by up to 60 times.
This article introduces cuDF, a Python GPU DataFrame library with a pandas‑like API, compares it to pandas, explains when to use cuDF versus Dask‑cuDF for single‑GPU or multi‑GPU workloads, and provides practical code examples for common data operations.
This article provides a comprehensive overview of NumPy and pandas, covering ndarray basics, multi‑dimensional array creation, core array attributes, broadcasting, random number generation, reshaping, as well as pandas Series and DataFrame structures, data import/export, grouping, merging, and advanced data manipulation techniques for scientific and data‑analysis tasks.
This article explains how Spark SQL processes joins—including inner, outer, semi, and anti joins—by describing the overall query planning flow, the three physical join strategies (sort‑merge, broadcast, and hash), and the specific implementation details for each join type.
This tutorial walks you through using Python and Pandas to import Excel files, explore DataFrames, apply filters, perform statistical calculations, create pivot tables, and emulate Excel functions like VLOOKUP, providing code examples and visual guides for each step.
This article provides an in‑depth technical overview of Apache Spark, covering its core concepts such as RDDs, transformation and action operations, execution models, Spark 2.0 enhancements like unified DataFrames/Datasets, whole‑stage code generation, Structured Streaming, and practical performance‑tuning guidance.
This article reviews Spark's evolution from version 1.2 to 1.6, explains the DataFrame and Tungsten projects, shares Hulu’s real‑world Spark deployments, and discusses performance‑related challenges such as stack overflow, streaming receiver latency, and class‑loader deadlocks.