Introduction to PySpark: Features, Core Components, Sample Code, and Use Cases

PySpark Introduction

PySpark is the Python API for Apache Spark, allowing developers to write Spark applications in Python for data cleaning, ETL, machine learning, and data analysis.

1. What is Spark?

Apache Spark is an open‑source, fast, unified engine for large‑scale data processing that supports batch, streaming, graph computation, and machine learning. Its main features are in‑memory computing (much faster than Hadoop MapReduce), a distributed computation framework that can handle TB‑PB data, and multi‑language support (Java, Scala, Python, R).

2. Advantages of PySpark

Feature

Description

Ease of use

Write in Python without needing Scala/Java.

Distributed computing power

Process massive data quickly.

Rich integration

Works with Hadoop, Hive, HDFS, Kafka, MySQL, etc.

Machine‑learning support

Provides MLlib for distributed ML tasks.

3. Core Components

SparkContext ( sc ) : Entry point that connects to a Spark cluster and creates RDDs.

RDD (Resilient Distributed Dataset) : Fundamental immutable distributed data collection.

DataFrame : Structured data abstraction similar to Pandas, recommended for most operations.

SparkSession ( spark ) : Unified entry for DataFrames and SQL, replaces older SQLContext/HiveContext.

Spark SQL : Enables SQL queries on DataFrames.

MLlib : Distributed machine‑learning library.

Structured Streaming : Real‑time stream processing.

4. Simple Example

<code>from pyspark.sql import SparkSession

# Create SparkSession
spark = SparkSession.builder \
    .appName("PySparkExample") \
    .getOrCreate()

# Create DataFrame
data = [("Alice", 21), ("Bob", 25), ("Cathy", 29)]
df = spark.createDataFrame(data, ["name", "age"])

# Operate on DataFrame
df.filter(df.age > 22).show()
</code>

Output:

<code>+-----+---+
| name|age|
+-----+---+
|  Bob| 25|
|Cathy| 29|
+-----+---+</code>

5. Typical Application Scenarios

Large‑scale log analysis

Data‑warehouse ETL processing

Real‑time stream data processing

Machine‑learning training and inference

Recommendation systems and behavior analysis in big‑data environments

6. PySpark vs Pandas

Aspect

PySpark

Pandas

Data scale

Big data (distributed)

Limited by single‑machine memory

Performance

Distributed high performance

Single‑threaded, slower

Learning curve

Medium

Easy

Typical scenario

Enterprise‑level big data analytics

Small‑scale data exploration

Further Learning Directions

Conversion and manipulation between RDD and DataFrame

SQL queries and data analysis with Spark SQL

Distributed machine‑learning model training

Integration with Hadoop and Hive

Structured Streaming for real‑time processing