In this section we learn about Python generators. They were introduced in Python 2.3. It is an easier way to create iterators using a keyword *yield* from a function.
>>> def my_generator():
... print "Inside my generator"
... yield 'a'
... yield 'b'
... yield 'c'
...
>>> my_generator()
<generator object my_generator at 0x7fbcfa0a6aa0>
In the above example we create a simple generator using the yield statements. We can use it in a for loop just like we use any other iterators.
>>> for char in my_generator():
... print char
...
Inside my generator
a
b
c
In the next example we will create the same Counter class using a generator function and use it in a for loop.
>>> def counter_generator(low, high):
... while low <= high:
... yield low
... low += 1
...
>>> for i in counter_generator(5,10):
... print i,
...
5 6 7 8 9 10
Inside the while loop when it reaches to the *yield* statement, the value of low is returned and the generator state is suspended. During the second *next* call the generator resumed where it freeze-ed before and then the value of *low* is increased by one. It continues with the while loop and comes to the *yield* statement again.
When you call an generator function it returns a *generator* object. If you call *dir* on this object you will find that it contains *__iter__* and *next* methods among the other methods.
>>> c = counter_generator(5,10)
>>> dir(c)
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__iter__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'close', 'gi_code', 'gi_frame', 'gi_running', 'next', 'send', 'throw']
We mostly use generators for laze evaluations. This way generators become a good approach to work with lots of data. If you don't want to load all the data in the memory, you can use a generator which will pass you each piece of data at a time.
One of the biggest example of such example is *os.path.walk()* function which uses a callback function and current *os.walk* generator. Using the generator implementation saves memory.
We can have generators which produces infinite values. The following is a one such example.
>>> def infinite_generator(start=0):
... while True:
... yield start
... start += 1
...
>>> for num in infinite_generator(4):
... print num,
... if num > 20:
... break
...
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
If we go back to the example of *my_generator* we will find one feature of generators. They are not re-usable.
>>> g = my_generator()
>>> for c in g:
... print c
...
Inside my generator
a
b
c
>>> for c in g:
... print c
...
One way to create a reusable generator is Object based generators which does not hold any state. Any class with a *__iter__* method which yields data can be used as a object generator. In the following example we will recreate out counter generator.
>>> class Counter(object):
... def __init__(self, low, high):
... self.low = low
... self.high = high
... def __iter__(self):
... counter = self.low
... while self.high >= counter:
... yield counter
... counter += 1
...
>>> gobj = Counter(5, 10)
>>> for num in gobj:
... print num,
...
5 6 7 8 9 10
>>> for num in gobj:
... print num,
...
5 6 7 8 9 10
