Concurrency & Memory Management

Modern computers have many cores, yet naive programs use one at a time. This chapter explains how to make Python work in parallel — with threads for I/O-bound work and processes for CPU-bound work — and how Python manages the memory those programs use. You will build a multithreaded web scraper and a multiprocessing data pipeline along the way.

Processes vs Threads

A process is a running program with its own memory space, its own file descriptors, and its own interpreter. A thread is a lightweight execution path inside a process; all threads of a process share the same memory.

• Processes are isolated: one crashing does not usually take others down, but starting one is expensive.
• Threads are cheap to start and can share data directly, but shared data needs careful locking.
• CPU-bound work (heavy math) benefits from processes, because Python threads are limited by the Global Interpreter Lock.
• I/O-bound work (waiting on networks, files, or databases) benefits from threads, because the waiting time can be used by other threads.

Multithreading with the threading Module

Creating and Starting Threads

import threading
import time

def work(name, delay):
    time.sleep(delay)
    print(f"thread {name} finished")

t1 = threading.Thread(target=work, args=("A", 2))
t2 = threading.Thread(target=work, args=("B", 1))

t1.start()
t2.start()
t1.join()
t2.join()
print("both threads done")

Race Conditions and Locks

When two threads mutate the same value without coordination, updates can be lost. A lock lets only one thread enter the protected section at a time.

import threading

balance = 100
lock = threading.Lock()

def withdraw(amount):
    global balance
    with lock:                       # acquire before touching balance
        if balance >= amount:
            balance -= amount

threads = [threading.Thread(target=withdraw, args=(30,)) for _ in range(5)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print("final balance:", balance)

Multiprocessing with the multiprocessing Module

Process and Pool

import multiprocessing as mp

def square(x):
    return x * x

if __name__ == "__main__":
    with mp.Pool(processes=4) as pool:
        results = pool.map(square, range(10))
    print(results)

Process launches one named task; Pool distributes many tasks over a fixed set of workers. Because each process has its own memory, results come back through queues or shared objects, not plain global variables.

Sharing State Between Processes

import multiprocessing as mp

def add_one(shared_value, lock):
    with lock:
        shared_value.value += 1

if __name__ == "__main__":
    lock = mp.Lock()
    counter = mp.Value("i", 0)
    procs = [mp.Process(target=add_one, args=(counter, lock)) for _ in range(8)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
    print("shared counter:", counter.value)

Thread and Process Pools via concurrent.futures

The concurrent.futures module offers a uniform interface: swap ThreadPoolExecutor for ProcessPoolExecutor and change almost nothing else.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def slow_add(a, b):
    return a + b

with ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(slow_add, range(5), range(5)))
print(results)

with ProcessPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(slow_add, range(5), range(5)))
print(results)

Use Case: Multithreaded Web Scraping

Fetching many pages is dominated by network wait, so threads are the right tool. Here each thread downloads a page's text with urllib and records its length.

import threading
import urllib.request

urls = ["https://example.com"] * 10
results = {}
lock = threading.Lock()

def fetch(url, idx):
    with urllib.request.urlopen(url) as resp:
        body = resp.read()
    with lock:
        results[idx] = len(body)

threads = [threading.Thread(target=fetch, args=(urls[i], i)) for i in range(len(urls))]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(sorted(results.items()))

Use Case: Multiprocessing for Number Crunching

Computing hashes of many files is CPU-bound, so processes give a real speedup. Each worker computes a checksum and returns it.

import hashlib
from concurrent.futures import ProcessPoolExecutor

def sha256_hex(text):
    return hashlib.sha256(text.encode()).hexdigest()

chunks = [f"sample-{i}" for i in range(20)]

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as pool:
        digests = list(pool.map(sha256_hex, chunks))
    print(digests[:3])

Memory Allocation, Deallocation, and Garbage Collection

Python manages memory automatically. When an object is created, space is allocated from a private heap; when no references point to it anymore, it becomes garbage.

Reference Counting and Generational GC

• Reference counting frees an object the moment its reference count hits zero — this is why many objects vanish immediately when you rebind a variable.
• Garbage collection handles reference cycles (two objects pointing at each other) that counting cannot see. The gc module runs collection in three generations; young objects are collected often, old ones rarely.

import gc
import sys

def make_cycle():
    a = {}
    b = {}
    a["other"] = b
    b["other"] = a
    return a, b

make_cycle()
collected = gc.collect()
print("objects collected:", collected)

Best Practices

• Prefer generators over giant lists when you only need one item at a time — they stream instead of filling memory.
• Release big data structures (del df or just reassign) when a pipeline stage no longer needs them.
• Work in chunks (Pandas chunksize, file line-by-line) for datasets larger than memory.
• Profile before optimizing: memory_profiler or the tracemalloc module show where memory actually goes.
• Close files, connections, and sockets with context managers so their resources are released promptly.