Master Python Multiprocessing: Fork, Process, Pool, and IPC Explained

Process

Note: This article is based on a Python 2.x environment.

Python implements multiprocessing mainly in two ways: using the os.fork function (Unix/Linux only) and using the cross‑platform multiprocessing module.

Using os.fork for multiprocessing

The Unix/Linux fork() system call creates a child process that is a copy of the parent; the child returns 0, the parent receives the child’s PID.

import os
print 'Process (%s) start...' % os.getpid()
pid = os.fork()
if pid == 0:
    print 'I am child process (%s) and my parent is %s.' % (os.getpid(), os.getppid())
else:
    print 'I (%s) just created a child process (%s).' % (os.getpid(), pid)

Creating processes with multiprocessing.Process

The Process class represents a process object; provide a target function and arguments, then start it with start() and synchronize with join().

# -*- coding:utf-8 -*-
from multiprocessing import Process
import os

def run_proc(name):
    print 'Run child process %s (%s)...' % (name, os.getpid())

if __name__ == '__main__':
    print 'Parent process %s.' % os.getpid()
    p = Process(target=run_proc, args=('test',))
    print 'Process will start.'
    p.start()
    p.join()
    print 'Process end.'

Using multiprocessing.Pool for a process pool

A Pool creates a pool of worker processes (default size equals CPU cores). Tasks are submitted with apply_async; the pool must be closed before joining.

# -*- coding:utf-8 -*-
from multiprocessing import Pool
import os, time, random

def long_time_task(name):
    print 'Run task %s (%s)...' % (name, os.getpid())
    start = time.time()
    time.sleep(random.random() * 3)
    end = time.time()
    print 'Task %s runs %0.2f seconds.' % (name, (end - start))

if __name__ == '__main__':
    print 'Parent process %s.' % os.getpid()
    p = Pool()
    for i in range(5):
        p.apply_async(long_time_task, args=(i,))
    print 'Waiting for all subprocesses done...'
    p.close()
    p.join()
    print 'All subprocesses done.'

Inter‑process communication

Python’s multiprocessing module wraps OS mechanisms and provides Queue and Pipe for data exchange. Queue is safe for multiple producers/consumers, while Pipe is suited for two‑process communication.

Queue communication

Use put to insert items and get to retrieve them, with optional blocked and timeout parameters.

# -*- coding:utf-8 -*-
from multiprocessing import Process, Queue
import os, time, random

def write(q):
    for value in ['A', 'B', 'C']:
        print 'Put %s to queue...' % value
        q.put(value)
        time.sleep(random.random())

def read(q):
    while True:
        value = q.get(True)
        print 'Get %s from queue.' % value

if __name__ == '__main__':
    q = Queue()
    pw = Process(target=write, args=(q,))
    pr = Process(target=read, args=(q,))
    pw.start()
    pr.start()
    pw.join()
    pr.terminate()

Pipe communication

Pipe()

returns two connection objects; with duplex=True both ends can send and receive.

import multiprocessing
import random, time, os

def proc_send(pipe, urls):
    for url in urls:
        print "process(%s) send:%s" % (os.getpid(), url)
        pipe.send(url)
        time.sleep(random.random())

def proc_recv(pipe):
    while True:
        print "Process(%s) rev:%s" % (os.getpid(), pipe.recv())
        time.sleep(random.random())

if __name__ == '__main__':
    pipe = multiprocessing.Pipe()
    p1 = multiprocessing.Process(target=proc_send, args=(pipe[0], ['url_'+str(i) for i in range(10)]))
    p2 = multiprocessing.Process(target=proc_recv, args=(pipe[1],))
    p1.start()
    p2.start()
    p1.join()
    p2.join()