Multiprocessing in Python | Python Course | Dataplexa

Multiprocessing in Python

In the last lesson you saw that Python's Global Interpreter Lock prevents threads from running CPU-bound code in true parallel. Multiprocessing is the solution. Instead of threads sharing one process, multiprocessing spawns entirely separate processes — each with its own Python interpreter, memory space, and GIL. True parallelism, full use of every CPU core.

This lesson covers Python's multiprocessing module, ProcessPoolExecutor, inter-process communication with Queue and Pipe, shared memory, and how to choose between threads and processes.

Processes vs Threads

Threads share memory within one process — lightweight, fast to create, limited by the GIL for CPU work.
Processes have completely separate memory — heavier to start, immune to the GIL, true parallelism across cores.
Use threads for I/O-bound work: downloading files, making API calls, reading databases.
Use processes for CPU-bound work: image processing, number crunching, ML preprocessing, compression.

Creating Processes

multiprocessing.Process works almost identically to threading.Thread — pass a target function, call start(), then join(). The critical difference is that each process runs in completely separate memory.

import multiprocessing
import time

def cpu_task(name, n):
    result = sum(i * i for i in range(n))
    print(f"[{name}] Result: {result:,}")

if __name__ == "__main__":   # required guard on Windows and macOS
    n = 5_000_000

    # Sequential — total time = sum of all durations
    start = time.perf_counter()
    cpu_task("Task-1", n)
    cpu_task("Task-2", n)
    cpu_task("Task-3", n)
    print(f"Sequential: {time.perf_counter() - start:.2f}s
")

    # Parallel — all three on separate CPU cores simultaneously
    start = time.perf_counter()
    processes = [
        multiprocessing.Process(target=cpu_task, args=(f"Task-{i}", n))
        for i in range(1, 4)
    ]
    for p in processes: p.start()
    for p in processes: p.join()
    print(f"Parallel: {time.perf_counter() - start:.2f}s")

    # Check exit codes
    for p in processes:
        print(f"{p.name} exited with code {p.exitcode}")

[Task-1] Result: 41,666,658,333,325,000 [Task-2] Result: 41,666,658,333,325,000 [Task-3] Result: 41,666,658,333,325,000 Sequential: 4.82s [Task-2] Result: 41,666,658,333,325,000 [Task-1] Result: 41,666,658,333,325,000 [Task-3] Result: 41,666,658,333,325,000 Parallel: 1.74s Process-1 exited with code 0 Process-2 exited with code 0 Process-3 exited with code 0

The if __name__ == "__main__": guard is required on Windows and macOS — without it, spawning processes causes infinite recursion as each child re-imports the module and tries to spawn again.
Each process gets a full copy of the program's memory at spawn time — changes in one process do not affect others.
p.exitcode is 0 on success, positive on error, or negative if killed by a signal.
Spawning processes is slower than creating threads — only worthwhile for tasks that take at least a fraction of a second.

ProcessPoolExecutor — The Modern Approach

concurrent.futures.ProcessPoolExecutor manages a pool of worker processes, distributes work automatically, and collects results cleanly — the recommended high-level API.

from concurrent.futures import ProcessPoolExecutor, as_completed
import time

def is_prime(n):
    if n < 2: return False
    for i in range(2, int(n**0.5) + 1):
        if n % i == 0: return False
    return True

def count_primes(start, end):
    return sum(1 for n in range(start, end) if is_prime(n))

if __name__ == "__main__":
    ranges = [(0, 250_000), (250_000, 500_000),
              (500_000, 750_000), (750_000, 1_000_000)]

    start = time.perf_counter()

    # ProcessPoolExecutor() defaults to number of CPU cores
    with ProcessPoolExecutor() as executor:
        futures = [executor.submit(count_primes, s, e) for s, e in ranges]
        results = [f.result() for f in futures]

    total = sum(results)
    print(f"Primes under 1,000,000: {total:,}")
    print(f"Time: {time.perf_counter() - start:.2f}s")

    # executor.map — simpler form, results in submission order
    with ProcessPoolExecutor() as executor:
        mapped = list(executor.map(count_primes,
                                   [s for s,e in ranges],
                                   [e for s,e in ranges]))
    print("Per-range counts:", mapped)

Primes under 1,000,000: 78,498 Time: 0.81s Per-range counts: [5761, 16252, 17082, 17403]

ProcessPoolExecutor() with no argument defaults to the number of CPU cores.
executor.submit(fn, *args) — schedules work and returns a Future; identical API to ThreadPoolExecutor.
executor.map(fn, *iterables) — simpler when you have parallel iterables; results in submission order.
The with block waits for all tasks and shuts down the pool cleanly.

Pool.map and Pool.starmap

For the simple pattern of applying one function to many inputs, multiprocessing.Pool is concise.

from multiprocessing import Pool

def square(n):
    return n * n

def power(base, exp):
    return base ** exp

if __name__ == "__main__":
    numbers = list(range(1, 13))

    with Pool() as pool:
        squares = pool.map(square, numbers)
    print("Squares:", squares)

    # starmap — function takes multiple arguments; each tuple is unpacked
    pairs = [(2, 10), (3, 5), (4, 4), (5, 3)]
    with Pool() as pool:
        powers = pool.starmap(power, pairs)
    print("Powers:", powers)

    # imap — lazy version; yields one result at a time (memory-efficient)
    with Pool() as pool:
        for result in pool.imap(square, range(5)):
            print(result, end="  ")
    print()

Squares: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144] Powers: [1024, 243, 256, 125] 0 1 4 9 16

pool.map(fn, iterable) — results in order; all held in memory at once.
pool.starmap(fn, iterable_of_tuples) — unpacks each tuple as separate arguments.
pool.imap(fn, iterable) — lazy; yields results one at a time — use for large datasets.
Always use with Pool() as pool: to ensure the pool is properly terminated.

Inter-Process Communication — Queue and Pipe

Processes cannot share variables directly — each has its own memory. Pass data between processes using Queue (multi-producer, multi-consumer) or Pipe (two-process point-to-point).

from multiprocessing import Process, Queue, Pipe

# Queue — safe communication between multiple processes
def producer(q, items):
    for item in items:
        q.put(item)
        print(f"Produced: {item}")
    q.put(None)   # sentinel — signals consumer to stop

def consumer(q):
    while True:
        item = q.get()
        if item is None: break
        print(f"Consumed: {item * 2}")

if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q, [10, 20, 30, 40]))
    p2 = Process(target=consumer, args=(q,))
    p1.start(); p2.start()
    p1.join();  p2.join()

    # Pipe — two-way connection between exactly two processes
    parent_conn, child_conn = Pipe()

    def child_work(conn):
        msg = conn.recv()
        conn.send(f"Echo: {msg}")
        conn.close()

    p = Process(target=child_work, args=(child_conn,))
    p.start()
    parent_conn.send("Hello from parent")
    print(parent_conn.recv())   # Echo: Hello from parent
    p.join()

Produced: 10 Produced: 20 Produced: 30 Produced: 40 Consumed: 20 Consumed: 40 Consumed: 60 Consumed: 80 Echo: Hello from parent

Queue.put(item) / Queue.get() — both are process-safe.
Use a sentinel value (like None) to signal consumers that no more items are coming.
Pipe() returns a pair of connection objects — faster than Queue for two-process communication.

Shared Memory — Value and Array

from multiprocessing import Process, Value, Array
import ctypes

def increment(counter, n):
    for _ in range(n):
        with counter.get_lock():   # process-safe lock
            counter.value += 1

def fill_array(arr, values):
    for i, v in enumerate(values):
        arr[i] = v

if __name__ == "__main__":
    # Value — shared scalar
    counter = Value(ctypes.c_int, 0)
    processes = [Process(target=increment, args=(counter, 50_000)) for _ in range(4)]
    for p in processes: p.start()
    for p in processes: p.join()
    print("Final counter:", counter.value)   # always 200,000

    # Array — shared fixed-size array
    arr = Array(ctypes.c_double, 5)
    p = Process(target=fill_array, args=(arr, [1.1, 2.2, 3.3, 4.4, 5.5]))
    p.start(); p.join()
    print("Shared array:", list(arr))

Final counter: 200,000 Shared array: [1.1, 2.2, 3.3, 4.4, 5.5]

Value(typecode, initial) — shared scalar; use C type codes like ctypes.c_int, ctypes.c_double.
Array(typecode, size_or_initializer) — shared fixed-size array.
Always use counter.get_lock() as a context manager when modifying shared values.
For anything more complex, use a Queue or multiprocessing.Manager.

Quick Reference Table

Factor	Threads	Processes
Best for	I/O-bound (network, disk, DB)	CPU-bound (computation, image processing)
Memory	Shared — fast, but needs locking	Separate — safe, but needs IPC
GIL impact	Limited by GIL for CPU work	Each process has its own GIL — true parallelism
Startup cost	Fast — milliseconds	Slower — tens of milliseconds
Communication	Direct shared variables (with locks)	Queue, Pipe, Value, Array
High-level API	ThreadPoolExecutor	ProcessPoolExecutor

Practice

Why does multiprocessing bypass the GIL where multithreading cannot?

Why is the if __name__ == "__main__": guard required when using multiprocessing?

What is the difference between pool.map() and pool.starmap()?

What tool do you use to safely pass data between two processes?

What lock method should you use when modifying a shared Value?

Which Pool method lazily yields results one at a time — useful for large datasets?

Quick Quiz

What is the key reason to use multiprocessing over multithreading for CPU-bound tasks?

Each process has its own GIL allowing true parallel execution across CPU cores
Processes start faster than threads
Processes share memory making communication easier

What does ProcessPoolExecutor() with no arguments default to?

The number of CPU cores on the machine
A single worker process
4 worker processes always

Why can processes not share variables directly the way threads can?

Each process has its own separate memory space — changes in one do not affect others
Processes run on different machines
The GIL prevents variable sharing between processes

What is the role of a sentinel value like None in a multiprocessing Queue?

To signal the consumer that no more items will be added and it should stop
To reset the queue back to empty
To pause the producer until the consumer catches up

When would you choose threads over processes for a performance-sensitive task?

When the task is I/O-bound — the GIL is released during I/O waits so threads provide real concurrency
When you need more than 4 parallel workers
When the task involves shared mutable state

What exit code does a process have after completing successfully?

0
1
None

NEXT UP

Working with APIs in Python

Making HTTP requests, handling responses, authentication, pagination, and building a reusable API client with the requests library.

← Previous Course Index Next →