So what are generators?
A generator is a special kind of function that returns an iterable set of values, one at a time which means it can be looped over to get the values one by one. It is also sometimes referred to as functions that can be 'paused'. In theory, generators sound quite complicated and confusing. It is best to explain using code examples. We hare already used a built-in Python generator
range to generate a range of values.
range_of_numbers = range(100) # a generator for num in range_of_numbers: print(num) # Since range is a generator, it can be iterated or looped
def my_infinite_generator(): num = 0 while True: yield num num +=1 result = my_infinite_generator() for i in result: print(i) # Keeps on printing values infinitely!
So what exactly is this generator function and how is it different from a normal function?
Functions can return only one value using the return statement. Once a function reaches a return statement, it returns the value and exits out. Whereas generator functions can return any number of values using a special keyword
yield. Whenever the
yield statement is reached, the function execution is paused and the control is passed to the whoever is calling the function.
The values of the generator can be extracted either by iteration or by using another built-in function called
next on the generator. In the above block of code, the values of the generator are printed using iteration (for loop). The same can be done manually by using the
def my_infinite_generator(): num = 0 while True: yield num num +=1 result = my_infinite_generator() print(next(result)) # 0 print(next(result)) # 1 print(next(result)) # 2 print(next(result)) # 3 print(next(result)) # 4
Note how the generator function is able to remember the value of num and is able to increment it. Whenever
yield statement is reached, it saves the local variables and their state and then the control is transferred to the caller.
def my_generator(max): num = 0 while num < 3: yield num num +=1 result = my_generator(3) print(next(result)) # 0 print(next(result)) # 1 print(next(result)) # 2 print(next(result)) # Stop Iteration Error
When an iteration is completed (in the above case once num = 3), the generator function automatically raises a StopIteration exception on further
next calls to it. If the same generator is iterated using a for loop, the loop will automatically stop once the StopIteration is raised by the generator. The for loop internally handles this and terminates.
Performance benefits of using generators
Generator functions are memory efficient. Generators really come in handy when dealing with large sets of data that requires lot of processing to compute results. Memory is a finite resource and our systems can only hold a limited amount of data in memory.
For instance, if we have to create a function that accepts a number and prints the Fibonacci sequence till that number, the conventional approach would be something like this
def fibonacci(num): sequence =  a,b = 0,1 for item in range(num): sequence.append(a) temp = a a = b b = temp + b return sequence result = fibonacci(20) print(result) # prints the fibonacci sequence
In the above function, the entire sequence is stored in memory as a list. The result is only printed once the entire sequence is stored in memory. This is usually fine when the number is small. However, if the number is large, the memory usage increases drastically and the process may even be killed if the memory overflows.
Using a generator the above function can be made more memory efficient
def fibonacci_generator(num): a,b = 0,1 for item in range(num): yield a temp = a a = b b = temp + b for num in fibonacci_generator(20): print(num)
In this case, only one value is stored in memory at a time, thus it is able to print values even if the provided number is very large.
Generators can be composed together or in other words, they can be piped together to combine results from different generators.
For instance, if we have to create a Fibonacci sequence with each value squared, they can be composed using two generator functions.
def fibonacci_generator(num): a,b = 0,1 for item in range(num): yield a temp = a a = b b = temp + b def square(nums): for num in nums: yield num**2 result = square(fibonacci_generator(5)) for num in result: print(num) # 0 1 1 4 9
This kind of composition can be very useful while computing large data sets and applying different operations on them in layers.
Tomorrow marks the start of the third week of this current Python challenge and as per my planned roadmap, I will be exploring another essential topic in Python - Modules. I am sure that will be quite interesting.
Have a great one!