
Python Course
Iterators in Python
Every time you write a for loop in Python, something is happening behind the scenes that most beginners never think about. Python is not simply stepping through a list — it is asking an object for its next value, one at a time, using a protocol built into the language itself. That protocol is the iterator protocol, and understanding it changes the way you think about loops, data, and memory.
This lesson unpacks exactly how iteration works, shows you the built-in tools that leverage it, teaches you to build your own iterable objects from scratch, and explains the important design choices that affect whether an iterator can be reused or not.
Iterables vs Iterators — The Key Distinction
These two terms are related but not the same, and confusing them is one of the most common sources of bugs for intermediate Python developers.
- An iterable is any object you can loop over — lists, tuples, strings, dictionaries, sets, files, and ranges are all iterables. An iterable knows how to produce an iterator when asked.
- An iterator is an object that does the actual work of stepping through values one at a time. It has internal state tracking where it currently is, and produces the next value on demand.
- Every iterator is also an iterable — but not every iterable is an iterator.
# The difference between an iterable and an iterator
nums = [10, 20, 30] # ITERABLE — can produce an iterator
# iter() asks the iterable for its iterator
it = iter(nums) # now 'it' is the ITERATOR
# next() asks the iterator for the next value
print(next(it)) # 10
print(next(it)) # 20
print(next(it)) # 30
# One more call raises StopIteration — nothing left
try:
print(next(it))
except StopIteration:
print("Exhausted — StopIteration raised")
# The original list is still intact — iterators are separate objects
print("List still intact:", nums)
print("List iterator type:", type(iter(nums)))
iter(obj)callsobj.__iter__()and returns the iterator.next(it)callsit.__next__()and returns the next value.- When values are exhausted,
StopIterationis raised — this is the normal signal Python uses to end aforloop. - The original list is never touched — the iterator is a separate, disposable object.
How a for Loop Really Works
Now that you know about iter() and next(), you can see exactly what Python does when it executes a for loop. This mental model is essential for understanding why certain things behave the way they do.
items = ["a", "b", "c"]
# What you write:
for item in items:
print(item)
print("---")
# What Python ACTUALLY does internally:
_it = iter(items) # Step 1: get the iterator
while True:
try:
item = next(_it) # Step 2: get next value
print(item) # Step 3: run loop body
except StopIteration:
break # Step 4: stop when exhausted
# This is why you can't loop over a plain integer:
try:
iter(42)
except TypeError as e:
print("Error:", e)
- Both versions produce identical output — the
forloop is syntactic sugar for exactly this pattern. StopIterationis not an error — it is the expected, normal signal that iteration is complete.- This model explains why
for n in 42fails — integers have no__iter__method. - It also explains why looping over a
map()orfilter()object works — they implement the iterator protocol.
The Iterator Protocol — __iter__ and __next__
Any object can become an iterator by implementing two special methods. This is the iterator protocol.
__iter__(self)— returns the iterator object itself (usually justreturn self)__next__(self)— returns the next value, or raisesStopIterationwhen done
By implementing these, your object works seamlessly in for loops, list(), sum(), zip(), and every other place Python expects an iterable.
# Custom iterator — counts down from start to 1
class Countdown:
"""Counts down from a given start number to 1."""
def __init__(self, start):
self.current = start
def __iter__(self):
return self # the object is its own iterator
def __next__(self):
if self.current <= 0:
raise StopIteration # signal completion
val = self.current
self.current -= 1 # advance internal state
return val
# Works in a for loop
for n in Countdown(5):
print(n, end=" ")
print()
# Works with list(), sum(), max()
print("As list:", list(Countdown(4)))
print("Sum :", sum(Countdown(4)))
print("Max :", max(Countdown(4)))
# next() with a default — avoids StopIteration exception
cd = Countdown(2)
print(next(cd, "done")) # 2
print(next(cd, "done")) # 1
print(next(cd, "done")) # "done" — default returned instead of raising
- Implementing
__iter__and__next__is all it takes — your class then works everywhere Python expects an iterable. next(iterator, default)— the two-argument form returns the default instead of raisingStopIteration. Very useful in production code.- Once exhausted, this iterator cannot be reused —
self.currentis now 0. Create a new instance to iterate again.
Separating the Iterable from the Iterator
When an iterator returns self from __iter__, it can only be traversed once. A cleaner design is to keep the data (iterable) and the traversal state (iterator) in separate classes. This lets you create multiple independent iterators over the same data simultaneously — exactly how Python's built-in types work.
# Separate iterable and iterator classes
class NumberRange:
"""The iterable — holds the data, creates fresh iterators on demand."""
def __init__(self, start, end):
self.start = start
self.end = end
def __iter__(self):
return NumberRangeIterator(self) # fresh iterator each time
class NumberRangeIterator:
"""The iterator — holds traversal state."""
def __init__(self, source):
self.current = source.start
self.end = source.end
def __iter__(self):
return self
def __next__(self):
if self.current > self.end:
raise StopIteration
val = self.current
self.current += 1
return val
r = NumberRange(1, 4)
# Two independent iterators over the same range
it1 = iter(r)
it2 = iter(r)
print(next(it1)) # 1 — it1 at position 1
print(next(it1)) # 2 — it1 advances
print(next(it2)) # 1 — it2 is independent, still at start
# The range itself can be looped multiple times
for n in r:
print(n, end=" ")
print()
for n in r: # second full loop — fresh iterator
print(n, end=" ")
print()
- When
__iter__returnsself, the object can only be traversed once — it is both iterable and iterator, but one-shot. - When
__iter__returns a fresh iterator object, multiple independent traversals are possible — this is howlist,range, anddictall work. - This explains why you can loop over a list twice but can only consume a
map()object once.
Checking for Iterability
You can check whether an object is iterable before attempting to loop over it — useful in functions that accept unknown inputs.
from collections.abc import Iterable, Iterator
# Check if something is iterable
for obj in [[1,2,3], "hello", 42, {"a":1}, range(5)]:
print(f"{str(obj):<20} iterable: {isinstance(obj, Iterable)}")
print()
# Check if something is an iterator (has __next__)
my_list = [1, 2, 3]
my_iter = iter(my_list)
print("list is Iterator :", isinstance(my_list, Iterator)) # False
print("iter is Iterator :", isinstance(my_iter, Iterator)) # True
print("list is Iterable :", isinstance(my_list, Iterable)) # True
print("iter is Iterable :", isinstance(my_iter, Iterable)) # True
collections.abc.Iterable— checks for__iter__.collections.abc.Iterator— checks for both__iter__and__next__.- A list is Iterable but not an Iterator — it can produce iterators but is not one itself.
Built-in Functions That Use Iterators
Once you understand the iterator protocol, you realise that Python's built-in functions work with any iterator — not just lists. This is what makes Python so composable and flexible.
class Squares:
"""Yields perfect squares from 1 up to limit."""
def __init__(self, limit):
self.n = 1
self.limit = limit
def __iter__(self): return self
def __next__(self):
if self.n > self.limit:
raise StopIteration
val = self.n ** 2
self.n += 1
return val
# All built-in functions work with custom iterators
print("list :", list(Squares(5)))
print("tuple :", tuple(Squares(4)))
print("sum :", sum(Squares(5)))
print("max :", max(Squares(5)))
print("min :", min(Squares(5)))
print("sorted:", sorted(Squares(5), reverse=True))
# zip with a custom iterator
names = ["Alice", "Bob", "Carol"]
squares = Squares(3)
for name, sq in zip(names, squares):
print(f" {name}: {sq}")
# enumerate with a custom iterator
print("Enumerated:")
for i, sq in enumerate(Squares(4), start=1):
print(f" {i}. {sq}")
list(),tuple(),set(),sum(),min(),max(),sorted(),enumerate(),zip()— all accept any iterator.- Each call consumes the iterator — create a new instance if you need to iterate again (or use the separate iterable/iterator design).
Lazy Iterators and Memory Efficiency
One of the biggest benefits of iterators is that they are lazy — they produce values one at a time, on demand, without generating the entire sequence in memory. This is critical when working with large data.
import sys
# A list of 1 million numbers — all in memory at once
big_list = list(range(1_000_000))
big_range = range(1_000_000) # range is lazy — stores only start/stop/step
print("list size:", sys.getsizeof(big_list), "bytes")
print("range size:", sys.getsizeof(big_range), "bytes") # tiny — ~48 bytes
# An infinite counter — impossible with a list, trivial with an iterator
class Counter:
"""An infinite counter starting from a given value."""
def __init__(self, start=0):
self.n = start
def __iter__(self): return self
def __next__(self):
val = self.n
self.n += 1
return val
# Take only the first 5 values using next()
c = Counter(10)
first_five = [next(c) for _ in range(5)]
print("First 5 from infinite counter:", first_five)
# Use itertools.islice to take a slice from any iterator
import itertools
c2 = Counter(100)
sample = list(itertools.islice(c2, 8))
print("8 from counter starting at 100:", sample)
range(1_000_000)uses just 48 bytes — it stores the formula, not the values.- Infinite iterators are perfectly valid — as long as you consume them with something that stops (
next(),islice(), a loop with abreak). itertools.islice(iterator, n)is the standard tool for taking a finite slice from any iterator — including infinite ones.
iter() with a Sentinel Value
There is a two-argument form of iter() that is less well known but extremely useful. iter(callable, sentinel) calls the callable repeatedly until it returns the sentinel value, then stops — no class or __next__ method needed.
import random
random.seed(42) # fixed seed for reproducible output
# Roll a die repeatedly until we get a 6
roller = iter(lambda: random.randint(1, 6), 6)
rolls = list(roller) # collect all rolls before the first 6
print("Rolls before 6:", rolls)
# Read a file in fixed-size chunks until empty bytes signal EOF
# (This is the most common real-world use)
# with open("large_file.bin", "rb") as f:
# for chunk in iter(lambda: f.read(4096), b""):
# process(chunk)
# Read lines from stdin until user types "quit"
# for line in iter(input, "quit"):
# print("You entered:", line)
# Count how many rolls to reach a 6 (re-roll)
random.seed(42)
count = 0
for _ in iter(lambda: random.randint(1, 6), 6):
count += 1
print(f"Took {count} rolls to get a 6")
iter(callable, sentinel)creates an iterator that callscallable()each timenext()is invoked.- When the callable returns the sentinel value,
StopIterationis raised automatically — the sentinel itself is never included in the results. - The file-chunking pattern is one of the most common uses in production — reading a binary file in 4KB chunks without loading the whole file.
Quick Reference Table
| Concept | What It Is | Key Method / Tool |
|---|---|---|
| Iterable | Any object you can loop over | __iter__() |
| Iterator | Object that yields values one at a time | __iter__() + __next__() |
iter(obj) | Gets the iterator from an iterable | Built-in function |
next(it) | Retrieves the next value | Built-in function |
next(it, default) | Next value or default (no exception) | Built-in function |
StopIteration | Normal signal that iteration is complete | Raised by __next__ |
| Iterator protocol | Contract: __iter__ + __next__ | Makes any class iterable |
| One-shot iterator | __iter__ returns self | Can only be traversed once |
| Reusable iterable | __iter__ returns a fresh iterator | Multiple traversals possible |
| Sentinel form | Call function until value is hit | iter(callable, sentinel) |
| Lazy evaluation | Values produced on demand — no full list in memory | Core benefit of iterators |
Practice
What built-in function do you call to get an iterator from an iterable?
What exception does __next__ raise to signal there are no more values?
What are the two special methods an object must implement for the iterator protocol?
What does the two-argument form iter(callable, sentinel) do?
Is every iterator also an iterable?
Which function from itertools lets you take a finite slice from an infinite iterator?
Quick Quiz
What does Python do internally at the start of every for loop?
What is the difference between an iterable and an iterator?
If you call next() on an exhausted iterator, what happens?
Why does separating the iterable from the iterator allow multiple independent traversals?
Which built-in function does NOT work with a custom iterator?
Why does range(1_000_000) use far less memory than list(range(1_000_000))?
yield keyword — and discover why generators are the preferred tool for working with large or infinite sequences.