Objects in the for loop: iterable, iterator and generator

In Python, we can iterate an object in for loop like this:

a = [1, 2, 3]
for i in a:
    print(i)

But not every object can be used like this. So what kind of object can be placed in a for-loop body? How can we make a custom class like this if some of its attributes can be retrieved sequentially. So in this article, I'd like to record my understanding about 3 confusing concepts in Python: iterable, iterator and generator.

Definition

"Iterable" sounds more like a conceptual term: we would say an object is iterable if its elements can be accessed sequentially, i.e., one-by-one, one-after-another. An iterable object must define the __iter__ dunder method which returns an itertor.

An iterator is also an object, which must contains two dunders: __iter__ (returns itself self) and __next__ returns the next elements to be accessed. Iterator serves as the proxy to retrieve the sequential elements of a container (e.g., list or dict) of a container (e.g., list or dict). Actually, the for loop in Python is just a syntactic sugar which automatically invokes the __next__ method of the iterator on behalf of programmer. Note that the iterators are disposables, that says, you cannot recycle or reuse them once the iteration terminates (see StopIteration). You have to obtain a new iterator from the corresponding container c by calling c.__iter__() dunder, or equivalently, iter(c).

Here is a simple example showing how iterator works:

class NumberIterator:
    def __init__(self, n):
        self.n = n
        self.now = n

    def __iter__(self):
        return self

    def __next__(self):
        if self.now > 0:
            x = self.now % 10
            self.now //= 10
            return x
        else:
            raise StopIteration()

for i in NumberIterator(114514):
    print(i)

It prints the digits of the given number in the reversed order, and an iterator needs to raise a StopIteration exception if no more items can be iterated.

Generator, on the other hand, serves a similar function as iterator to produce data one after another. In CPython, there are two ways to declare a generator:

1. generator expression ¹

sum(x*x for x in range(10))

Compare to list comprehension, generator expression does not create the substantial object (i.e., list in this example), instead it just evaluates the necessary values one at a time. This saves the required memory to evaluate the statement. It actually creates a anomynous generator function (see below) in current scope.

2. function with yield ²

def fib():
    a, b = 0, 1
    while 1:
       yield b
       a, b = b, a+b

In Python, a function with the keyword yield is another type of generator. It allows the interpreter to interrupt the execution of function at the place of yield, and resumes later from there. This makes it possible to implement a coroutine or fine-grained overlapped pipeline execution control of programs.

Dive into CPython, what happens when a for loop encouters an iterator

As aforementioned, an iterable will be casted to iterator through its __iter__ dunder in the for loop statement. So when interpreter evaluates the for loop with an iterator, what happens?

To be continued...