Master Python Multiprocessing: From Basics to Real-World File Copying

1. Process Introduction

A process is a running program consisting of the program code, data, and a process control block; it is the basic unit of resource scheduling.

A program is static code that is not executing.

2. Comparison between Threads and Processes

Process: can handle multiple tasks; multiple applications can run simultaneously.

Thread: can handle multiple tasks within a single application, e.g., multiple chat windows in QQ.

Fundamental difference: a process is the basic unit of OS resource allocation, while a thread is the basic unit of task scheduling and execution.

Advantages of using multiple processes:

1. Independent GIL: each process has its own Global Interpreter Lock, allowing true multi‑core utilization.

2. Higher efficiency: for CPU‑bound tasks, multiprocessing offers significant speedup compared to multithreading.

3. Implementing Multiprocessing in Python

Example using the Process class:

import multiprocessing

def process(index):
    print(f'Process: {index}')

if __name__ == '__main__':
    for i in range(5):
        p = multiprocessing.Process(target=process, args=(i,))
        p.start()

Note: args must be a tuple; even a single argument requires a trailing comma.

Running this prints the process indices 0‑4.

Example by subclassing Process:

from multiprocessing import Process
import time

class MyProcess(Process):
    def __init__(self, loop):
        Process.__init__(self)
        self.loop = loop

    def run(self):
        for count in range(self.loop):
            time.sleep(1)
            print(f'Pid:{self.pid} LoopCount: {count}')

if __name__ == '__main__':
    for i in range(2, 5):
        p = MyProcess(i)
        p.start()

Note: the execution logic must be placed in the run method; start() triggers it.

4. Inter‑process Communication

Using a Queue:

from multiprocessing import Queue
import multiprocessing

def download(p):
    lst = [11, 22, 33, 44]
    for item in lst:
        p.put(item)
    print('Data downloaded....')

def savedata(p):
    lst = []
    while True:
        data = p.get()
        lst.append(data)
        if p.empty():
            break
    print(lst)

def main():
    p1 = Queue()
    t1 = multiprocessing.Process(target=download, args=(p1,))
    t2 = multiprocessing.Process(target=savedata, args=(p1,))
    t1.start()
    t2.start()

if __name__ == '__main__':
    main()

Result shows the downloaded list.

Global variables are not shared across processes:

import multiprocessing

a = 1

def demo1():
    global a
    a += 1

def demo2():
    print(a)

def main():
    t1 = multiprocessing.Process(target=demo1)
    t2 = multiprocessing.Process(target=demo2)
    t1.start()
    t2.start()

if __name__ == '__main__':
    main()

Output is 1, demonstrating that globals are not shared.

5. Process Pool Communication

Using Pool to manage many tasks:

from multiprocessing import Pool
import os, time, random

def worker(a):
    t_start = time.time()
    print(f'{a} starts, pid {os.getpid()}')
    time.sleep(random.random()*2)
    t_stop = time.time()
    print(a, "completed, time %.2f" % (t_stop - t_start))

if __name__ == '__main__':
    po = Pool(3)
    for i in range(10):
        po.apply_async(worker, (i,))
    print("--start--")
    po.close()
    po.join()
    print("--end--")

Only three processes run concurrently; new tasks are added as slots free up.

6. Case Study: Batch File Copying

Workflow:

Obtain the source folder name.

Create a new destination folder.

List all files in the source folder.

Create a process pool.

Add copy tasks to the pool.

Key functions:

import multiprocessing
import os
import time

def copy_file(Q, oldfolderName, newfolderName, file_name):
    time.sleep(0.5)
    old_file = open(oldfolderName + '/' + file_name, 'rb')
    content = old_file.read()
    old_file.close()
    new_file = open(newfolderName + '/' + file_name, 'wb')
    new_file.write(content)
    new_file.close()
    Q.put(file_name)

def main():
    oldfolderName = input('Enter folder to copy:')
    newfolderName = oldfolderName + 'Copy'
    if not os.path.exists(newfolderName):
        os.mkdir(newfolderName)
    filenames = os.listdir(oldfolderName)
    pool = multiprocessing.Pool(5)
    Q = multiprocessing.Manager().Queue()
    for file_name in filenames:
        pool.apply_async(copy_file, args=(Q, oldfolderName, newfolderName, file_name))
    pool.close()
    copy_file_num = 0
    file_count = len(filenames)
    while True:
        file_name = Q.get()
        copy_file_num += 1
        time.sleep(0.2)
        print(f'Copy progress {copy_file_num*100/file_count:.2f} %', end='')
        if copy_file_num >= file_count:
            break

if __name__ == '__main__':
    main()

The script displays a progress bar and copies all files from the source to the destination folder.