Generators

In the last lesson you built iterators by writing a full class with __iter__ and __next__. It worked — but it required a lot of boilerplate for something conceptually simple. Generators are Python's answer to that problem. With a single keyword — yield — you can turn any function into an iterator instantly, with no class, no state tracking, and no StopIteration management needed.

Generators are one of the most powerful and memory-efficient tools in Python. This lesson covers generator functions, generator expressions, the yield from delegation syntax, and the practical scenarios where generators outperform every other approach.

The yield Keyword

A regular function runs to completion and returns one value. A generator function uses yield instead of return. Each time yield is reached, the function pauses, hands the value to the caller, and freezes its entire state — local variables, the instruction pointer, everything. The next call to next() resumes exactly where it left off.

Why it exists: producing values lazily — one at a time, only when needed — means you never have to build the entire sequence in memory. A generator that produces a million numbers uses almost no memory, because it only ever holds one number at a time.

Real-world use: a data pipeline reads a multi-gigabyte log file line by line using a generator, processing each line without loading the whole file into RAM.

# A simple generator function using yield

def count_up(start, end):
    n = start
    while n <= end:
        yield n      # pause here, hand n to the caller, freeze state
        n += 1       # resumes HERE on the next next() call

gen = count_up(1, 4)   # calling the function returns a generator object
print(type(gen))        # 

print(next(gen))   # 1
print(next(gen))   # 2
print(next(gen))   # 3
print(next(gen))   # 4

# Works in a for loop just like any iterator
for n in count_up(1, 4):
    print(n, end=" ")

• Calling a generator function does not run any of its code — it returns a generator object
• Code runs only when next() is called — one step at a time
• When the function body ends naturally, StopIteration is raised automatically
• The generator object satisfies the full iterator protocol — use it anywhere an iterator is expected

Generator vs Regular Function — The Memory Difference

The most important practical advantage of generators is memory. A list comprehension builds every value and stores them all at once. A generator produces values one at a time and discards each after use.

# Memory comparison — list vs generator

import sys

# List comprehension — builds ALL values in memory immediately
big_list = [x ** 2 for x in range(1_000_000)]
print("List size:", sys.getsizeof(big_list), "bytes")

# Generator expression — holds only ONE value at a time
big_gen = (x ** 2 for x in range(1_000_000))
print("Generator size:", sys.getsizeof(big_gen), "bytes")

• The list holds one million integers in memory — the generator holds almost nothing
• If you only need to iterate once and do not need random access, a generator is almost always the better choice
• Both produce the same values when consumed — the difference is entirely in memory usage

Generator Expressions

Just as list comprehensions have a compact one-line syntax, generators have generator expressions. The syntax is identical to a list comprehension but uses parentheses () instead of square brackets [].

# Generator expressions — lazy one-liners

# List comprehension — eager, builds everything now
squares_list = [x ** 2 for x in range(1, 6)]

# Generator expression — lazy, produces values on demand
squares_gen = (x ** 2 for x in range(1, 6))

print(squares_list)          # [1, 4, 9, 16, 25]
print(list(squares_gen))     # [1, 4, 9, 16, 25] — same values

# Generators compose well — pass directly into functions
total = sum(x ** 2 for x in range(1, 6))   # no extra () needed inside sum()
print("Sum of squares:", total)

# Filter with a condition
evens = (x for x in range(1, 11) if x % 2 == 0)
print(list(evens))

• Use () for a generator expression, [] for a list comprehension
• When passing a generator expression as the only argument to a function, the outer () can be omitted
• Generator expressions are ideal when you only need to iterate once and do not need the full list

Infinite Generators

Because generators produce values lazily, they can represent sequences that never end — something impossible with a list. You simply never let the generator run out of values.

Real-world use: an event loop generator produces a continuous stream of sensor readings or timestamps, and the calling code decides when to stop consuming.

# An infinite generator — never raises StopIteration on its own

def integers_from(n):
    """Yields every integer starting from n, forever."""
    while True:
        yield n
        n += 1

gen = integers_from(1)

# Take only the first 5 values — we control when to stop
for _ in range(5):
    print(next(gen), end=" ")
print()

# Use itertools.islice to safely slice an infinite generator
import itertools
first_ten = list(itertools.islice(integers_from(1), 10))
print(first_ten)

• Never call list() or sum() on an infinite generator — it will run forever
• Use next() manually, a for loop with a break, or itertools.islice() to take a fixed number of values
• itertools is the standard library module built specifically for working with iterators and generators

Chaining Generators — Pipelines

Generators compose naturally into pipelines — a chain of lazy transformations where each step yields to the next. No intermediate lists are created, and memory stays flat no matter how many stages are in the chain.

Real-world use: a data processing pipeline reads a file, strips whitespace, filters blank lines, and parses each line — all lazily, all in a chain of generators.

# Generator pipeline — lazy, zero intermediate lists

def read_numbers(data):
    """Yields each number from a list (simulates reading a data source)."""
    for n in data:
        yield n

def square(numbers):
    """Yields the square of each number."""
    for n in numbers:
        yield n ** 2

def only_even(numbers):
    """Yields only even numbers."""
    for n in numbers:
        if n % 2 == 0:
            yield n

# Build the pipeline — nothing runs yet
raw      = read_numbers(range(1, 11))
squared  = square(raw)
filtered = only_even(squared)

# Values are pulled through the entire pipeline only here
print(list(filtered))

• Each generator in the chain pulls from the previous one — values flow through lazily
• Only the final consumer (list() here) triggers actual computation
• Adding more stages to the pipeline does not increase memory usage

yield from — Delegating to a Sub-generator

yield from lets a generator delegate part of its work to another iterable or generator. Instead of looping and yielding each value manually, you hand control to the sub-generator entirely.

# yield from — delegate to another iterable

def flatten(nested):
    """Flatten a list of lists into a single stream of values."""
    for sublist in nested:
        yield from sublist   # delegate — yields every item in sublist

data = [[1, 2, 3], [4, 5], [6, 7, 8, 9]]
print(list(flatten(data)))

# yield from also works with any iterable — strings, ranges, etc.
def alphabet_then_digits():
    yield from "abc"
    yield from range(1, 4)

print(list(alphabet_then_digits()))

• yield from iterable is equivalent to for item in iterable: yield item — but cleaner and faster
• Works with any iterable — lists, ranges, strings, or other generators
• Particularly useful for recursively flattening nested structures

Summary Table

Practice Questions

Quiz

Next up — Working with JSON: reading, writing, and transforming JSON data in Python.