Multithreading in Python

Most programs run one instruction at a time — sequentially. But many real-world tasks spend most of their time waiting: waiting for a file to load, for a network response, for a database query to return. Multithreading lets your program do other work during those waits by running multiple threads of execution concurrently within the same process.

This lesson covers Python's threading module, thread synchronisation, the Global Interpreter Lock, and the practical patterns you will use when building programs that need to handle multiple tasks at once.

What is a Thread?

A thread is a unit of execution within a process. All threads in a process share the same memory space — the same variables, objects, and files. This makes communication between threads easy but introduces the risk of two threads modifying the same data at the same time, causing unpredictable results.

• A Python program always starts with one thread — the main thread
• You can create additional threads to run functions concurrently
• Threads are lightweight compared to processes — creating one is fast and uses little memory
• Threads are best suited for I/O-bound tasks — tasks that spend time waiting for external resources

Creating and Starting Threads

The threading.Thread class creates a new thread. Pass it a target function and optional args or kwargs, then call .start() to begin execution.

Real-world use: a web scraper downloads ten pages concurrently — each page download runs in its own thread so they all happen in parallel instead of one after another.

# Creating and starting threads

import threading
import time

def download(url, duration):
    print(f"[{threading.current_thread().name}] Starting: {url}")
    time.sleep(duration)   # simulate network wait
    print(f"[{threading.current_thread().name}] Done: {url}")

# Sequential — total time = sum of all durations
start = time.perf_counter()
download("page_a.html", 2)
download("page_b.html", 1)
download("page_c.html", 3)
print(f"Sequential: {time.perf_counter() - start:.2f}s\n")

# Concurrent — total time ≈ longest single duration
start = time.perf_counter()
threads = [
    threading.Thread(target=download, args=("page_a.html", 2), name="T1"),
    threading.Thread(target=download, args=("page_b.html", 1), name="T2"),
    threading.Thread(target=download, args=("page_c.html", 3), name="T3"),
]
for t in threads:
    t.start()
for t in threads:
    t.join()    # wait for all threads to finish before continuing
print(f"Concurrent: {time.perf_counter() - start:.2f}s")

• t.start() begins execution of the thread — the main thread continues immediately
• t.join() blocks the calling thread until the target thread finishes — essential for collecting results
• Always join threads before accessing any results they produced
• The order of thread output is non-deterministic — threads run in whatever order the OS schedules them

The Global Interpreter Lock (GIL)

Python has a mechanism called the Global Interpreter Lock (GIL) — a mutex that allows only one thread to execute Python bytecode at a time, even on multi-core hardware. This is the most important thing to understand about Python threads.

• I/O-bound tasks — threads work well. When a thread is waiting for I/O (network, disk, database), it releases the GIL, allowing other threads to run. This is why multithreading gives real speedups for I/O-bound work.
• CPU-bound tasks — threads do not help and can even be slower. Because only one thread runs Python code at a time, multiple threads compete for the GIL rather than running in true parallel. Use multiprocessing (next lesson) for CPU-bound tasks instead.

# GIL illustration — threads help I/O-bound, not CPU-bound

import threading
import time

def cpu_task(n):
    """CPU-bound — pure computation, holds the GIL."""
    total = 0
    for i in range(n):
        total += i * i
    return total

def io_task(seconds):
    """I/O-bound — releases the GIL while sleeping."""
    time.sleep(seconds)

# I/O-bound: threading gives real speedup
start = time.perf_counter()
threads = [threading.Thread(target=io_task, args=(1,)) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()
print(f"5 I/O tasks with threads: {time.perf_counter() - start:.2f}s")  # ~1s

# CPU-bound: threads offer no speedup due to GIL
start = time.perf_counter()
threads = [threading.Thread(target=cpu_task, args=(2_000_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(f"4 CPU tasks with threads: {time.perf_counter() - start:.2f}s")  # no improvement

Thread Synchronisation — Lock

Because threads share memory, two threads can read and modify the same variable simultaneously — producing incorrect results. A Lock ensures only one thread accesses a critical section of code at a time.

Real-world use: a multi-threaded web server increments a request counter on every hit. Without a lock, some increments get lost — two threads read the same value before either writes back.

# Race condition — and how to fix it with a Lock

import threading

counter = 0

def increment_unsafe():
    global counter
    for _ in range(100_000):
        counter += 1   # read-modify-write — not atomic, unsafe

def increment_safe(lock):
    global counter
    for _ in range(100_000):
        with lock:          # only one thread enters this block at a time
            counter += 1

# Unsafe — race condition produces wrong result
counter = 0
threads = [threading.Thread(target=increment_unsafe) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()
print("Unsafe result:", counter)   # likely less than 500,000

# Safe — Lock prevents race condition
counter = 0
lock = threading.Lock()
threads = [threading.Thread(target=increment_safe, args=(lock,)) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()
print("Safe result:  ", counter)   # always exactly 500,000

• threading.Lock() creates a lock object — call lock.acquire() to lock and lock.release() to unlock
• Use with lock: — the context manager form — to guarantee the lock is always released even if an exception occurs
• A thread that tries to acquire an already-held lock will block until the lock is released
• Keep locked sections as short as possible to avoid slowing down other threads unnecessarily

ThreadPoolExecutor — The Modern Approach

concurrent.futures.ThreadPoolExecutor is the high-level, modern way to run functions in a thread pool. It manages thread creation, reuse, and result collection for you — much cleaner than managing threads manually.

Real-world use: making dozens of API calls concurrently — submit them all to the pool and collect results as they complete, without manually creating and joining threads.

# ThreadPoolExecutor — high-level thread pool

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def fetch(url):
    time.sleep(1)   # simulate a 1-second API call
    return f"Response from {url}"

urls = [
    "https://api.example.com/users",
    "https://api.example.com/orders",
    "https://api.example.com/products",
    "https://api.example.com/reports",
]

start = time.perf_counter()

# Submit all tasks and collect results as they complete
with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(fetch, url): url for url in urls}
    for future in as_completed(futures):
        print(future.result())

print(f"All done in {time.perf_counter() - start:.2f}s")

• executor.submit(fn, *args) schedules a function and returns a Future object immediately
• as_completed(futures) yields futures in the order they finish — not the order they were submitted
• executor.map(fn, iterable) is a simpler alternative when you want results in submission order
• The with block waits for all submitted tasks to complete before exiting — automatic join

Daemon Threads

A daemon thread runs in the background and is killed automatically when the main thread exits. Use daemon threads for background tasks that should not prevent the program from shutting down.

# Daemon threads — background tasks that don't block program exit

import threading
import time

def background_monitor():
    while True:
        print("[Monitor] Checking system health...")
        time.sleep(2)

# daemon=True — thread dies when main thread exits
monitor = threading.Thread(target=background_monitor, daemon=True)
monitor.start()

print("Main thread doing work...")
time.sleep(3)
print("Main thread done — program exits, daemon thread is killed")

• Set daemon=True before calling .start()
• Non-daemon threads keep the program alive until they finish — the program will not exit while a non-daemon thread is running
• Use daemon threads for log writers, health monitors, background sync tasks

Summary Table

Practice Questions

Quiz

Next up — Multiprocessing: bypassing the GIL for true parallel CPU-bound execution using separate processes.