Multithreading in Python | Python Course | Dataplexa

Multithreading in Python

Most programs run one instruction at a time — sequentially. But many real-world tasks spend most of their time waiting: for a network response, a file load, a database query. Multithreading lets your program do other work during those waits by running multiple threads concurrently within the same process.

This lesson covers Python's threading module, thread synchronisation with locks, the Global Interpreter Lock, the ThreadPoolExecutor, and daemon threads.

What is a Thread?

A thread is a unit of execution within a process. All threads share the same memory — the same variables, objects, and files. This makes communication easy but introduces the risk of two threads modifying the same data simultaneously, causing unpredictable results.

  • A Python program always starts with one thread — the main thread.
  • Threads are lightweight — creating one is fast and uses little memory.
  • Threads are best for I/O-bound tasks — tasks that spend time waiting for external resources.
  • For CPU-bound tasks, use multiprocessing instead (next lesson).

Creating and Starting Threads

threading.Thread creates a thread. Pass it a target function and optional args, then call .start().

import threading
import time

def download(url, duration):
    print(f"[{threading.current_thread().name}] Starting: {url}")
    time.sleep(duration)   # simulate network wait
    print(f"[{threading.current_thread().name}] Done: {url}")

# Sequential — total time = sum of all durations
start = time.perf_counter()
download("page_a.html", 2)
download("page_b.html", 1)
download("page_c.html", 3)
print(f"Sequential: {time.perf_counter() - start:.2f}s
")

# Concurrent — total time ≈ longest single duration
start = time.perf_counter()
threads = [
    threading.Thread(target=download, args=("page_a.html", 2), name="T1"),
    threading.Thread(target=download, args=("page_b.html", 1), name="T2"),
    threading.Thread(target=download, args=("page_c.html", 3), name="T3"),
]
for t in threads: t.start()
for t in threads: t.join()    # wait for all to finish
print(f"Concurrent: {time.perf_counter() - start:.2f}s")
[MainThread] Starting: page_a.html [MainThread] Done: page_a.html [MainThread] Starting: page_b.html [MainThread] Done: page_b.html [MainThread] Starting: page_c.html [MainThread] Done: page_c.html Sequential: 6.00s [T1] Starting: page_a.html [T2] Starting: page_b.html [T3] Starting: page_c.html [T2] Done: page_b.html [T1] Done: page_a.html [T3] Done: page_c.html Concurrent: 3.00s
  • t.start() begins the thread — the main thread continues immediately without waiting.
  • t.join() blocks the calling thread until the target thread finishes — always join before using results.
  • Thread output order is non-deterministic — the OS schedules threads however it likes.

The Global Interpreter Lock (GIL)

Python has a Global Interpreter Lock — a mutex that allows only one thread to execute Python bytecode at a time, even on multi-core hardware.

  • I/O-bound tasks: threads work well. When a thread waits for I/O, it releases the GIL, letting other threads run — giving real speedups.
  • CPU-bound tasks: threads do not help and can be slower. Only one thread runs Python code at a time, so threads compete for the GIL rather than running truly in parallel. Use multiprocessing for CPU-bound tasks.
import threading, time

def cpu_task(n):
    """CPU-bound — holds the GIL while computing."""
    return sum(i * i for i in range(n))

def io_task(seconds):
    """I/O-bound — releases the GIL while sleeping."""
    time.sleep(seconds)

# I/O-bound: real speedup from threading
start = time.perf_counter()
threads = [threading.Thread(target=io_task, args=(1,)) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()
print(f"5 I/O tasks with threads: {time.perf_counter() - start:.2f}s")   # ~1.0s

# CPU-bound: no speedup — GIL prevents true parallelism
start = time.perf_counter()
threads = [threading.Thread(target=cpu_task, args=(2_000_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(f"4 CPU tasks with threads: {time.perf_counter() - start:.2f}s")   # no improvement
5 I/O tasks with threads: 1.00s 4 CPU tasks with threads: 2.41s

Thread Synchronisation — Lock

Because threads share memory, two threads can read and modify the same variable simultaneously — producing incorrect results. A Lock ensures only one thread accesses a critical section at a time.

import threading

counter = 0

def increment_unsafe():
    global counter
    for _ in range(100_000):
        counter += 1   # read-modify-write — NOT atomic, unsafe under threads

def increment_safe(lock):
    global counter
    for _ in range(100_000):
        with lock:          # only one thread here at a time
            counter += 1

# Unsafe — race condition → wrong result
counter = 0
threads = [threading.Thread(target=increment_unsafe) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()
print("Unsafe:", counter)    # typically much less than 500,000

# Safe — Lock prevents race condition
counter = 0
lock = threading.Lock()
threads = [threading.Thread(target=increment_safe, args=(lock,)) for _ in range(5)]
for t in threads: t.start()
for t in threads: t.join()
print("Safe:  ", counter)    # always exactly 500,000
Unsafe: 412,847 Safe: 500,000
  • Use with lock: — the context manager form guarantees the lock is released even on exception.
  • A thread that tries to acquire an already-held lock blocks until it is released.
  • Keep locked sections short — holding a lock too long slows other threads unnecessarily.
  • threading.RLock() is a re-entrant lock — a thread can acquire it multiple times without deadlocking itself.

ThreadPoolExecutor — The Modern Approach

concurrent.futures.ThreadPoolExecutor manages thread creation, reuse, and result collection for you — the clean high-level API for running functions in a thread pool.

from concurrent.futures import ThreadPoolExecutor, as_completed
import time

def fetch(url):
    time.sleep(1)   # simulate a 1-second API call
    return f"Response from {url}"

urls = [
    "https://api.example.com/users",
    "https://api.example.com/orders",
    "https://api.example.com/products",
    "https://api.example.com/reports",
]

start = time.perf_counter()

with ThreadPoolExecutor(max_workers=4) as executor:
    # submit() returns Future objects immediately
    futures = {executor.submit(fetch, url): url for url in urls}

    # as_completed yields futures as they finish (not submission order)
    for future in as_completed(futures):
        url  = futures[future]
        result = future.result()
        print(f"{url.split('/')[-1]}: {result[:30]}")

print(f"All done in {time.perf_counter() - start:.2f}s")

# executor.map() alternative — simpler, results in submission order
print("
Using map:")
with ThreadPoolExecutor(max_workers=4) as executor:
    for result in executor.map(fetch, urls):
        print(result[:40])
orders: Response from https://api.example.com/orders users: Response from https://api.example.com/users products: Response from https://api.example.com/products reports: Response from https://api.example.com/reports All done in 1.01s Using map: Response from https://api.example.com/users Response from https://api.example.com/orders Response from https://api.example.com/products Response from https://api.example.com/reports
  • executor.submit(fn, *args) schedules a function and returns a Future immediately.
  • as_completed(futures) yields futures in finish order — not submission order.
  • executor.map(fn, iterable) — simpler alternative when you want results in submission order.
  • The with block automatically waits for all tasks to complete before exiting.

Daemon Threads

A daemon thread runs in the background and is killed automatically when the main thread exits. Use daemon threads for background tasks that should not prevent shutdown.

import threading, time

def background_monitor():
    while True:
        print("[Monitor] Checking system health...")
        time.sleep(2)

# daemon=True — thread dies automatically when main thread exits
monitor = threading.Thread(target=background_monitor, daemon=True, name="HealthMonitor")
monitor.start()

print(f"Main thread working... (monitor: {monitor.name}, daemon={monitor.daemon})")
time.sleep(3)
print("Main thread done — daemon thread is killed automatically")
Main thread working... (monitor: HealthMonitor, daemon=True) [Monitor] Checking system health... [Monitor] Checking system health... Main thread done — daemon thread is killed automatically
  • Set daemon=True before calling .start().
  • Non-daemon threads keep the program alive until they finish — the process will not exit while any non-daemon thread is running.
  • Use daemon threads for log writers, health monitors, and background sync tasks.

Quick Reference Table

ToolPurposeKey Usage
threading.ThreadCreate and run a threadThread(target=fn, args=(...))
t.start()Begin thread executionCall after creating the thread
t.join()Wait for thread to finishCall before using thread results
threading.LockPrevent race conditionswith lock:
threading.RLockRe-entrant lockWhen the same thread may acquire the lock twice
ThreadPoolExecutorHigh-level thread poolexecutor.submit(fn, *args)
Daemon threadBackground task killed on exitThread(..., daemon=True)

Practice

What method starts a thread's execution after it is created?



What does t.join() do?



What is the GIL and which type of task does it prevent from running in true parallel?



What is a race condition and how does a Lock prevent it?



What happens to a daemon thread when the main thread exits?



Which function from concurrent.futures yields futures in the order they finish?



Quick Quiz

For which type of task does Python multithreading provide the most benefit?





Why does the GIL make multithreading ineffective for CPU-bound tasks?





What is the preferred way to use a Lock?





What does as_completed(futures) yield?





What prevents a program from exiting while a non-daemon thread is still running?





Which lock type allows the same thread to acquire it multiple times without deadlocking itself?





NEXT UP
Multiprocessing in Python
Bypassing the GIL for true parallel execution using separate processes — the right tool for CPU-bound work.