Overview

iterate means to repeat something over and over again

exactly what we've done with loops.

An iterator is an object that knows how to return items from a collection, one at a time, while keeping track of its current position within that collection.

An iterable is an object, which has method for returning iterator.

iterator in Python and the __next__ method

iterator is any object, implementing the __next__ method

When __next__ method is called, the iterator should return its "next value".

If the iterator has no more values to return, it should raise a StopIteration exception

There is a built-in function called next, wrapping around __next__ that you can use for convenience.

next(it) is equivalent to it.__next__()

Iterator protocol in Python2 and Python3

in Python 2 an iterator object must have a method called next()

in Python 3 an iterator object must have a method called __next__()

iterable in Python and the __iter__ method

itarable is an object, implementing the __iter__ method

The __iter__ method returns an iterator.

The built-in function iter() can also be used to get an iterator from iterable.

Every sequence in Python is iterable. But there are other iterable types, which are not sequences, like sets.

iterable vs iterator

An iterable is any object, that can return an iterator

Iterator is any object that returns its next value.

Iterables can represent finite as well as infinite source of data

iterables and iterators - examples

for loop over iterable:

      # strings are iterable in Python:
      s = "abc"

      # iterate over the string iterable with for loop
      for i in s:
          print(i, end=" ")

      # OUTPUT
      # a
      # b
      # c
    

for loop behaviour using iter() and next():

      # strings are iterable in Python:
      s = "abc"

      # lets get an iterator from the string iterable:
      s_iterator = iter(s)

      #ask the string iterator for values
      print( next(s_iterator) )
      print( next(s_iterator) )
      print( next(s_iterator) )

      # OUTPUT
      # a
      # b
      # c
    

Custom Iterator Types

A class that wants to be an iterator should implement the two methods:

__next__() - that implement the Iterator Protocol.

__iter__() - method that returns self

Which means that in Python an Iterator is also an Iterable!

This requirement is to make iterators to behave like sequences; in particular so that code receiving an iterator can use a for-loop over the iterator.

check Python API Specification in PEP 234 -- Iterators for more details

Custom Iterator Types - simplest example

      class SimplestNumbersIterator():
        """Simplest Number Iterator, which can iterate on only two values - 1 and 2."""
        def __init__(self):
          self.num = 0
          self.max = 2

        def __iter__(self):
          # make the iterator iterable, as well
          return self

        def __next__(self):
          # generate a number to return
          if self.num < self.max:
            self.num += 1
            return self.num
          else:
            raise StopIteration
    

Custom Iterator on Fibonacci Sequence - example

      class Fibs:
        def __init__(self, limit):
          self.a = 0
          self.b = 1
          self.limit = limit
        def __next__(self):
          self.a, self.b = self.b, self.a + self.b
          if ( self.a < self.limit):
            return self.a
          else:
            raise StopIteration
        def __iter__(self):
          return self

      fib_numbers = FibsIterable(30)

      for i in fib_numbers:
        print( i )
    

Resources

Iterator Types @Python3 docs

PEP 234 -- Iterators

Overview

A Generator is a function that works as a factory for iterators.

Python’s generators provide a convenient way to implement the iterator protocol.

Or more formally: a Python generator is a function which returns an iterator by calling yield

Any function that contains a yield statement can be called a generator!

"lazy evaluation"

Generators are the primary means by which Python implements so called lazy evaluation

The lazy (on demand) generation of values translates to lower memory usage and (most of the times) in a performance improvement

How a generator function works?

When a generator function is called, it returns an generator-iterator, that will controls the execution of the generator function.

When the __next__ method of that generator is called, the execution of the function proceeds to the first yield expression, and the execution is "freezed" - the current state of the function is preserved, and the yield value and the he control is transferred to the caller.

When calling again the __next__ method of that generator, the exection of the function continues from where it was freezed, and continues as described above.

If calling __next__, the execution code of the function contrains no yield statement - then the StopIteration error is raised.

Making a Generator

      def foo_generator():
        yield 1
        yield 2

      foo_gen = foo_generator()

      print( next(foo_gen) )
      print( next(foo_gen) )

      # OUTPUT
      # 1
      # 2
    

Custom Generator vs Custom Iterator - example

The Iterator way:

      class SimplestNumbersIterator():
        """Simplest Number Iterator, which can iterate on only two values - 1 and 2.

          Disclaimer: Use it just to trace how __next__ and __iter__ operators works.This example is not useful in real world!
        """
        def __init__(self):
          self.num = 0
          self.max = 2

        def __iter__(self):
          return self

        def __next__(self):
          # generate a number to return
          if self.num < self.max:
            self.num += 1
            print("Iterate on", self.num)
            return self.num
          else:
            raise StopIteration
    

Play with the full example @repl.it: Custom-Generator-vs-Custom-Iterator

Custom Generator vs Custom Iterator - example

The Generator way:

      def simplest_numbers_generator():
        """Simplest Number Generator, yielding only two values - 1 and 2.

          Disclaimer: Use it just to trace how next and yield operators works.
                      This example is not useful in real world!
        """
        num = 1
        print("Yielding", num)
        yield num

        num +=1
        print("Yielding", num)
        yield 2

        num +=1
        print("Yielding", num)
        print("Sory, there is nothing to yield...")
    

Play with the full example @repl.it: Custom-Generator-vs-Custom-Iterator

Example - even numbers generator

      #define the generator function:
      def numbers_generator(start,end):
        num  = start
        while num<=end:
          # yield is almost like return, but it freezes the execution
          yield num
          num += 1

      my_numbers = numbers_generator(1,10)
      # iterate over our generator:
      for i in my_numbers:
        print(i)
      

Example - nested list flatten generator

      nested = [[1, 2], [3, 4], [5]]

      def flatten(nested):
        for sublist in nested:
          for element in sublist:
            yield element

      print( list(flatten(nested)) )
    

Example - iterator for random "name" like Cyrillic strings

Naive generation of random "name" like Cyrillic strings

Resources

PEP 255 -- Simple Generators

Generators @python wiki

The Task

Write a generator function, such that will yield an even number in pre-given interval

      def even_numbers_generator(start,end):
        # write your code here

      for i in even_numbers_generator(1,10):
        print(i)

      ### expected result
      #2
      #4
      #6
      #8
      #10
    

The Task

Define a generator function which will yield a Cyrillic letter, starting from 'А', to 'Я'

You can get a letter form its code, using the chr() built-in function, as shown in next example

      print( chr(1040) )
      # 'А''

      print( chr(1041) )
      # 'Б'

      print( chr(1070) )
      # 'Ю'
      print( chr(1071) )
      # 'Я'