Iterators in Python | Python Course | Dataplexa

Iterators in Python

Every time you write a for loop in Python, something is happening behind the scenes that most beginners never think about. Python is not simply stepping through a list — it is asking an object for its next value, one at a time, using a protocol built into the language itself. That protocol is the iterator protocol, and understanding it changes the way you think about loops, data, and memory.

This lesson unpacks exactly how iteration works, shows you the built-in tools that leverage it, teaches you to build your own iterable objects from scratch, and explains the important design choices that affect whether an iterator can be reused or not.

Iterables vs Iterators — The Key Distinction

These two terms are related but not the same, and confusing them is one of the most common sources of bugs for intermediate Python developers.

  • An iterable is any object you can loop over — lists, tuples, strings, dictionaries, sets, files, and ranges are all iterables. An iterable knows how to produce an iterator when asked.
  • An iterator is an object that does the actual work of stepping through values one at a time. It has internal state tracking where it currently is, and produces the next value on demand.
  • Every iterator is also an iterable — but not every iterable is an iterator.
# The difference between an iterable and an iterator

nums = [10, 20, 30]   # ITERABLE — can produce an iterator

# iter() asks the iterable for its iterator
it = iter(nums)       # now 'it' is the ITERATOR

# next() asks the iterator for the next value
print(next(it))   # 10
print(next(it))   # 20
print(next(it))   # 30

# One more call raises StopIteration — nothing left
try:
    print(next(it))
except StopIteration:
    print("Exhausted — StopIteration raised")

# The original list is still intact — iterators are separate objects
print("List still intact:", nums)
print("List iterator type:", type(iter(nums)))
10 20 30 Exhausted — StopIteration raised List still intact: [10, 20, 30] List iterator type: <class 'list_iterator'>
  • iter(obj) calls obj.__iter__() and returns the iterator.
  • next(it) calls it.__next__() and returns the next value.
  • When values are exhausted, StopIteration is raised — this is the normal signal Python uses to end a for loop.
  • The original list is never touched — the iterator is a separate, disposable object.

How a for Loop Really Works

Now that you know about iter() and next(), you can see exactly what Python does when it executes a for loop. This mental model is essential for understanding why certain things behave the way they do.

items = ["a", "b", "c"]

# What you write:
for item in items:
    print(item)

print("---")

# What Python ACTUALLY does internally:
_it = iter(items)            # Step 1: get the iterator
while True:
    try:
        item = next(_it)     # Step 2: get next value
        print(item)          # Step 3: run loop body
    except StopIteration:
        break                # Step 4: stop when exhausted

# This is why you can't loop over a plain integer:
try:
    iter(42)
except TypeError as e:
    print("Error:", e)
a b c --- a b c Error: 'int' object is not iterable
  • Both versions produce identical output — the for loop is syntactic sugar for exactly this pattern.
  • StopIteration is not an error — it is the expected, normal signal that iteration is complete.
  • This model explains why for n in 42 fails — integers have no __iter__ method.
  • It also explains why looping over a map() or filter() object works — they implement the iterator protocol.

The Iterator Protocol — __iter__ and __next__

Any object can become an iterator by implementing two special methods. This is the iterator protocol.

  • __iter__(self) — returns the iterator object itself (usually just return self)
  • __next__(self) — returns the next value, or raises StopIteration when done

By implementing these, your object works seamlessly in for loops, list(), sum(), zip(), and every other place Python expects an iterable.

# Custom iterator — counts down from start to 1

class Countdown:
    """Counts down from a given start number to 1."""

    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self            # the object is its own iterator

    def __next__(self):
        if self.current <= 0:
            raise StopIteration    # signal completion
        val = self.current
        self.current -= 1          # advance internal state
        return val

# Works in a for loop
for n in Countdown(5):
    print(n, end=" ")
print()

# Works with list(), sum(), max()
print("As list:", list(Countdown(4)))
print("Sum    :", sum(Countdown(4)))
print("Max    :", max(Countdown(4)))

# next() with a default — avoids StopIteration exception
cd = Countdown(2)
print(next(cd, "done"))   # 2
print(next(cd, "done"))   # 1
print(next(cd, "done"))   # "done" — default returned instead of raising
5 4 3 2 1 As list: [4, 3, 2, 1] Sum : 10 Max : 4 2 1 done
  • Implementing __iter__ and __next__ is all it takes — your class then works everywhere Python expects an iterable.
  • next(iterator, default) — the two-argument form returns the default instead of raising StopIteration. Very useful in production code.
  • Once exhausted, this iterator cannot be reused — self.current is now 0. Create a new instance to iterate again.

Separating the Iterable from the Iterator

When an iterator returns self from __iter__, it can only be traversed once. A cleaner design is to keep the data (iterable) and the traversal state (iterator) in separate classes. This lets you create multiple independent iterators over the same data simultaneously — exactly how Python's built-in types work.

# Separate iterable and iterator classes

class NumberRange:
    """The iterable — holds the data, creates fresh iterators on demand."""
    def __init__(self, start, end):
        self.start = start
        self.end   = end

    def __iter__(self):
        return NumberRangeIterator(self)   # fresh iterator each time

class NumberRangeIterator:
    """The iterator — holds traversal state."""
    def __init__(self, source):
        self.current = source.start
        self.end     = source.end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        val = self.current
        self.current += 1
        return val

r = NumberRange(1, 4)

# Two independent iterators over the same range
it1 = iter(r)
it2 = iter(r)

print(next(it1))   # 1  — it1 at position 1
print(next(it1))   # 2  — it1 advances
print(next(it2))   # 1  — it2 is independent, still at start

# The range itself can be looped multiple times
for n in r:
    print(n, end=" ")
print()
for n in r:          # second full loop — fresh iterator
    print(n, end=" ")
print()
1 2 1 1 2 3 4 1 2 3 4
  • When __iter__ returns self, the object can only be traversed once — it is both iterable and iterator, but one-shot.
  • When __iter__ returns a fresh iterator object, multiple independent traversals are possible — this is how list, range, and dict all work.
  • This explains why you can loop over a list twice but can only consume a map() object once.

Checking for Iterability

You can check whether an object is iterable before attempting to loop over it — useful in functions that accept unknown inputs.

from collections.abc import Iterable, Iterator

# Check if something is iterable
for obj in [[1,2,3], "hello", 42, {"a":1}, range(5)]:
    print(f"{str(obj):<20} iterable: {isinstance(obj, Iterable)}")

print()

# Check if something is an iterator (has __next__)
my_list  = [1, 2, 3]
my_iter  = iter(my_list)

print("list is Iterator :", isinstance(my_list, Iterator))   # False
print("iter is Iterator :", isinstance(my_iter, Iterator))   # True
print("list is Iterable :", isinstance(my_list, Iterable))   # True
print("iter is Iterable :", isinstance(my_iter, Iterable))   # True
[1, 2, 3] iterable: True hello iterable: True 42 iterable: False {'a': 1} iterable: True range(0, 5) iterable: True list is Iterator : False iter is Iterator : True list is Iterable : True iter is Iterable : True
  • collections.abc.Iterable — checks for __iter__.
  • collections.abc.Iterator — checks for both __iter__ and __next__.
  • A list is Iterable but not an Iterator — it can produce iterators but is not one itself.

Built-in Functions That Use Iterators

Once you understand the iterator protocol, you realise that Python's built-in functions work with any iterator — not just lists. This is what makes Python so composable and flexible.

class Squares:
    """Yields perfect squares from 1 up to limit."""
    def __init__(self, limit):
        self.n     = 1
        self.limit = limit
    def __iter__(self): return self
    def __next__(self):
        if self.n > self.limit:
            raise StopIteration
        val   = self.n ** 2
        self.n += 1
        return val

# All built-in functions work with custom iterators
print("list  :", list(Squares(5)))
print("tuple :", tuple(Squares(4)))
print("sum   :", sum(Squares(5)))
print("max   :", max(Squares(5)))
print("min   :", min(Squares(5)))
print("sorted:", sorted(Squares(5), reverse=True))

# zip with a custom iterator
names   = ["Alice", "Bob", "Carol"]
squares = Squares(3)
for name, sq in zip(names, squares):
    print(f"  {name}: {sq}")

# enumerate with a custom iterator
print("Enumerated:")
for i, sq in enumerate(Squares(4), start=1):
    print(f"  {i}. {sq}")
list : [1, 4, 9, 16, 25] tuple : (1, 4, 9) sum : 55 max : 25 min : 1 sorted: [25, 16, 9, 4, 1] Alice: 1 Bob: 4 Carol: 9 Enumerated: 1. 1 2. 4 3. 9 4. 16
  • list(), tuple(), set(), sum(), min(), max(), sorted(), enumerate(), zip() — all accept any iterator.
  • Each call consumes the iterator — create a new instance if you need to iterate again (or use the separate iterable/iterator design).

Lazy Iterators and Memory Efficiency

One of the biggest benefits of iterators is that they are lazy — they produce values one at a time, on demand, without generating the entire sequence in memory. This is critical when working with large data.

import sys

# A list of 1 million numbers — all in memory at once
big_list  = list(range(1_000_000))
big_range = range(1_000_000)    # range is lazy — stores only start/stop/step

print("list  size:", sys.getsizeof(big_list),  "bytes")
print("range size:", sys.getsizeof(big_range), "bytes")   # tiny — ~48 bytes

# An infinite counter — impossible with a list, trivial with an iterator
class Counter:
    """An infinite counter starting from a given value."""
    def __init__(self, start=0):
        self.n = start
    def __iter__(self): return self
    def __next__(self):
        val    = self.n
        self.n += 1
        return val

# Take only the first 5 values using next()
c = Counter(10)
first_five = [next(c) for _ in range(5)]
print("First 5 from infinite counter:", first_five)

# Use itertools.islice to take a slice from any iterator
import itertools
c2     = Counter(100)
sample = list(itertools.islice(c2, 8))
print("8 from counter starting at 100:", sample)
list size: 8000056 bytes range size: 48 bytes First 5 from infinite counter: [10, 11, 12, 13, 14] 8 from counter starting at 100: [100, 101, 102, 103, 104, 105, 106, 107]
  • range(1_000_000) uses just 48 bytes — it stores the formula, not the values.
  • Infinite iterators are perfectly valid — as long as you consume them with something that stops (next(), islice(), a loop with a break).
  • itertools.islice(iterator, n) is the standard tool for taking a finite slice from any iterator — including infinite ones.

iter() with a Sentinel Value

There is a two-argument form of iter() that is less well known but extremely useful. iter(callable, sentinel) calls the callable repeatedly until it returns the sentinel value, then stops — no class or __next__ method needed.

import random

random.seed(42)   # fixed seed for reproducible output

# Roll a die repeatedly until we get a 6
roller = iter(lambda: random.randint(1, 6), 6)
rolls  = list(roller)    # collect all rolls before the first 6
print("Rolls before 6:", rolls)

# Read a file in fixed-size chunks until empty bytes signal EOF
# (This is the most common real-world use)
# with open("large_file.bin", "rb") as f:
#     for chunk in iter(lambda: f.read(4096), b""):
#         process(chunk)

# Read lines from stdin until user types "quit"
# for line in iter(input, "quit"):
#     print("You entered:", line)

# Count how many rolls to reach a 6 (re-roll)
random.seed(42)
count = 0
for _ in iter(lambda: random.randint(1, 6), 6):
    count += 1
print(f"Took {count} rolls to get a 6")
Rolls before 6: [1, 5, 3, 2, 4] Took 5 rolls to get a 6
  • iter(callable, sentinel) creates an iterator that calls callable() each time next() is invoked.
  • When the callable returns the sentinel value, StopIteration is raised automatically — the sentinel itself is never included in the results.
  • The file-chunking pattern is one of the most common uses in production — reading a binary file in 4KB chunks without loading the whole file.

Quick Reference Table

ConceptWhat It IsKey Method / Tool
IterableAny object you can loop over__iter__()
IteratorObject that yields values one at a time__iter__() + __next__()
iter(obj)Gets the iterator from an iterableBuilt-in function
next(it)Retrieves the next valueBuilt-in function
next(it, default)Next value or default (no exception)Built-in function
StopIterationNormal signal that iteration is completeRaised by __next__
Iterator protocolContract: __iter__ + __next__Makes any class iterable
One-shot iterator__iter__ returns selfCan only be traversed once
Reusable iterable__iter__ returns a fresh iteratorMultiple traversals possible
Sentinel formCall function until value is hititer(callable, sentinel)
Lazy evaluationValues produced on demand — no full list in memoryCore benefit of iterators

Practice

What built-in function do you call to get an iterator from an iterable?



What exception does __next__ raise to signal there are no more values?



What are the two special methods an object must implement for the iterator protocol?



What does the two-argument form iter(callable, sentinel) do?



Is every iterator also an iterable?



Which function from itertools lets you take a finite slice from an infinite iterator?



Quick Quiz

What does Python do internally at the start of every for loop?






What is the difference between an iterable and an iterator?





If you call next() on an exhausted iterator, what happens?





Why does separating the iterable from the iterator allow multiple independent traversals?





Which built-in function does NOT work with a custom iterator?





Why does range(1_000_000) use far less memory than list(range(1_000_000))?





NEXT UP
Generators in Python
Learn a far simpler way to build iterators using the yield keyword — and discover why generators are the preferred tool for working with large or infinite sequences.