Python Essentials: CSV Handling, itertools, collections, Debugging, and Performance Tips

1. Common Tools

1.1 Read/Write CSV Files

<code>import csv
# read/write without header
with open(name, 'rt', encoding='utf-8', newline='') as f:
    for row in csv.reader(f):
        print(row[0], row[1])
with open(name, mode='wt') as f:
    f_csv = csv.writer(f)
    f_csv.writerow(['symbol', 'change'])

# read/write with header
with open(name, mode='rt', newline='') as f:
    for row in csv.DictReader(f):
        print(row['symbol'], row['change'])
with open(name, mode='wt') as f:
    header = ['symbol', 'change']
    f_csv = csv.DictWriter(f, header)
    f_csv.writeheader()
    f_csv.writerow({'symbol': xx, 'change': xx})
</code>

When a CSV file is very large, Python may raise _csv.Error: field larger than field limit (131072) ; increase the limit with:

<code>import sys
csv.field_size_limit(sys.maxsize)
</code>

CSV can also read tab‑separated data:

<code>f = csv.reader(f, delimiter='\t')
</code>

1.2 Iterator Tools

The itertools module provides many iterator utilities, such as slicing, filtering, and combinatorial generators:

<code>import itertools
itertools.islice(iterable, start=None, stop, step=None)
itertools.filterfalse(predicate, iterable)
itertools.takewhile(predicate, iterable)
itertools.dropwhile(predicate, iterable)
itertools.compress(iterable, selectors)
sorted(iterable, key=None, reverse=False)
itertools.groupby(iterable, key=None)
itertools.permutations(iterable, r=None)
itertools.combinations(iterable, r=None)
itertools.combinations_with_replacement(...)
itertools.chain(*iterables)
import heapq
heapq.merge(*iterables, key=None, reverse=False)
zip(*iterables)
itertools.zip_longest(*iterables, fillvalue=None)
</code>

1.3 Counter

The collections.Counter class counts occurrences of elements in an iterable:

<code>import collections
counter = collections.Counter(iterable)
frequency = counter[key]
most_common = counter.most_common(n=None)
counter.update(iterable)
counter1 + counter2
counter1 - counter2
collections.Counter(list1) == collections.Counter(list2)
</code>

1.4 defaultdict (Dict with Default Value)

defaultdict automatically creates a default value for missing keys:

<code>import collections
dd = collections.defaultdict(type)  # type is called with no arguments when a new key is accessed
</code>

1.5 OrderedDict (Ordered Dictionary)

OrderedDict preserves insertion order when iterating:

<code>import collections
od = collections.OrderedDict(items=None)
</code>

2. High‑Performance Programming and Debugging

2.1 Output Errors and Warnings

Write to standard error:

<code>import sys
sys.stderr.write('error message')
</code>

Emit warnings with the warnings module:

<code>import warnings
warnings.warn(message, category=UserWarning)
# categories: DeprecationWarning, SyntaxWarning, RuntimeWarning, ResourceWarning, FutureWarning
</code>

Control warning display from the command line:

<code>$ python -W all     # show all warnings
$ python -W ignore  # ignore all warnings
$ python -W error   # turn warnings into exceptions
</code>

2.2 In‑Code Testing

Use the built‑in __debug__ flag for debug‑only code, which is stripped when Python runs with the -O (optimize) flag:

<code># debug block
if __debug__:
    print('debug info')
# run with optimization
$ python -O main.py
</code>

2.3 Code Style Checking

Run pylint to catch style and syntax issues before execution:

<code>pylint main.py
</code>

2.4 Measuring Execution Time

Profile a script with cProfile :

<code>$ python -m cProfile main.py
</code>

Measure a specific code block using a context manager:

<code>from contextlib import contextmanager
from time import perf_counter

@contextmanager
def timeblock(label):
    tic = perf_counter()
    try:
        yield
    finally:
        toc = perf_counter()
        print(f"{label} : {toc - tic}")

with timeblock('counting'):
    # code to time
    pass
</code>

General performance‑optimisation principles:

Focus on bottlenecks, not the whole program.

Avoid global variables; local look‑ups are faster (15‑30% speed‑up).

Prefer local references over attribute access; use from module import name when possible.

Use built‑in data structures (str, list, set, dict) implemented in C.

Minimise creation of unnecessary intermediate objects and deep copies.

Prefer ':'.join([...]) over repeated string concatenation; also consider using print(..., sep=':') for output.

3. Additional Python Tricks

3.1 argmin and argmax

<code>items = [2, 1, 3, 4]
argmin = min(range(len(items)), key=items.__getitem__)
# argmax works analogously
</code>

3.2 Transpose a 2‑D List

<code>A = [['a11', 'a12'], ['a21', 'a22'], ['a31', 'a32']]
A_transpose = list(zip(*A))               # list of tuples
A_transpose = [list(col) for col in zip(*A)]  # list of lists
</code>

3.3 Split a 1‑D List into a 2‑D List

<code>A = [1, 2, 3, 4, 5, 6]
# Preferred method
result = list(zip(*[iter(A)] * 2))
</code>

For more Python resources, scan the QR code below to receive free course materials, e‑books, tutorials, and source code collections.