Python iteration and the for loop¶

I have spent some time really digging into generators and by extension iterables. The official docs are pretty good, but I found that David Beazley's generator explanation fills a lot of the gaps.

To truly understand generators you must start with the concept of protocols. So, I decided this might be a good time for a little article about iteration and look inside the python for loop.

Protocols¶

One of the nice features of Python is that under the hood objects can be designed and implemented by so-called protocols. These are not very well documented, but several exist:

• Descriptor
• Iteration
• Pickling
• Sequence

These protocols are implemented by providing your own special methods, sometimes called dunders.

So, to make your object iterable it must respond to the iteration protocol, namely the __iter__() and next() methods. [1]

Iteration and generators¶

Have you ever wondered how the for loop actually worked in Python? Surprise, it's all based on the same iteration protocol.

David Beazley's generator explanation does a great job of explaining how this protocol fuels the implementation of the for loop:

_iter = iter(obj)               # Get iterator object
while 1
    try:
        x = _iter.next()       # Get next item
    except StopIteration:  # No more items
        break
    # statements

Make sure to read through the entire generator presentation. There are all sorts of other nice details you might have forgotten or never seen before.

[1] next() doesn't have a __next__() in Python 2.7, but this was fixed in Python 3.