1 views collected around this technical thread.
This article introduces PySpark as the Python API for Apache Spark, explains Spark's core concepts and advantages, details PySpark's main components and a simple code example, compares it with Pandas, and outlines typical big‑data scenarios and further learning directions.
This article introduces ten highly practical Python libraries for data analysis—from Pandas and NumPy for data manipulation to Matplotlib, Seaborn, Plotly, Bokeh for visualization, and Scikit‑learn, Prophet, Dask, and PySpark for machine learning and big‑data processing—each illustrated with concise code snippets.
This article presents a comprehensive overview of Spark 3.4, covering community growth statistics, major SQL improvements such as default column values and timestamp handling, new PySpark and streaming capabilities, and the emerging AI ecosystem that integrates natural‑language interfaces and Spark AI services.
This article describes how Tubi’s data science team built a robust KPI forecasting system with Facebook’s Prophet, covering visualization dashboards, anomaly detection, feature engineering, PySpark deployment, and evaluation using Brier scores to improve business decision‑making.
This tutorial explains several techniques for adding new columns to PySpark DataFrames—including native Spark functions, user‑defined functions, RDD transformations, Pandas UDFs, and SQL queries—while demonstrating data loading, schema handling, and code examples for each method.
This tutorial demonstrates how to initialize a SparkContext in PySpark, perform simple parallel computations such as temperature conversion and reduction, create a SparkSession to read CSV data, and apply common DataFrame operations like selecting columns, adding new columns, filtering, grouping, and aggregating.
This article explains PySpark's multi‑process architecture, how the Python driver uses Py4J to call Java/Scala APIs, the implementation of RDD and DataFrame interfaces, executor‑side process communication and serialization with Arrow, and the design of Pandas UDFs, while also discussing current limitations and future directions.