Python Data Analysis Techniques: List Comprehensions, Lambda, Map/Filter, NumPy, and Pandas Operations

This article introduces eight Python data analysis methods that improve runtime efficiency and make code more elegant.

One‑Line List Definition (List Comprehension)

Instead of using a verbose for‑loop to build a list, Python’s list comprehension can create the same list in a single line.

x = [1, 2, 3, 4]
out = []
for item in x:
    out.append(item**2)
print(out)  # [1, 4, 9, 16]

# vs.

x = [1, 2, 3, 4]
out = [item**2 for item in x]
print(out)  # [1, 4, 9, 16]

Lambda Expressions

Lambda provides a quick way to define small, anonymous functions.

double = lambda x: x * 2
print(double(5))  # 10

Map and Filter

Combining lambda with map applies a transformation to each element, while filter selects elements that satisfy a condition.

# Map example
seq = [1, 2, 3, 4, 5]
result = list(map(lambda var: var*2, seq))
print(result)  # [2, 4, 6, 8, 10]

# Filter example
result = list(filter(lambda x: x > 2, seq))
print(result)  # [3, 4, 5]

NumPy: arange and linspace

arange

generates an arithmetic sequence with a specified step, while linspace creates a specified number of evenly spaced points between start and stop.

# arange(start, stop, step)
np.arange(3, 7, 2)  # array([3, 5])

# linspace(start, stop, num)
np.linspace(2.0, 3.0, num=5)
# array([2.0, 2.25, 2.5, 2.75, 3.0])

Axis in Pandas

The axis parameter determines whether an operation works on rows (0) or columns (1).

df.drop('Column A', axis=1)   # drop a column
df.drop('Row A', axis=0)      # drop a row
df.shape  # (number_of_rows, number_of_columns)

Concat, Merge, and Join

These functions combine DataFrames in different ways, similar to SQL operations.

Concat : Append DataFrames vertically or horizontally.

Merge : Join on matching keys.

Join : Merge based on index or column names without specifying a key.

Apply

apply

runs a function on each element of a Series or across a DataFrame axis, avoiding explicit loops.

df = pd.DataFrame([[4, 9]] * 3, columns=['A', 'B'])
print(df.apply(np.sqrt))
print(df.apply(np.sum, axis=0))
print(df.apply(np.sum, axis=1))

Pivot Table

Pandas’ pivot_table creates spreadsheet‑style pivot tables for quick data summarisation.

# Basic pivot by manager and rep
pd.pivot_table(df, index=['Manager', 'Rep'])

# Pivot with specific values
pd.pivot_table(df, index=['Manager', 'Rep'], values=['Price'])

The examples demonstrate useful Python functions and concepts for efficient data manipulation and analysis.