Join Bill Weinman for an in-depth discussion in this video Using generators, part of Python 3 Essential Training.
- View Offline
A generator object is an object that can be used in the context of an iterable, like for instance in a for loop. Let's take a look at how this is done in Python. We'll make a working copy of generator .py. I'll call it generator-working.py. We'll open that up and we see that here we have a range object being used in the context of an iterator. So o is getting assigned to this range object that's a range of 25 and it's being used in the context of this for loop which is an iterable context and if I run this, you see that we get series of numbers from 0 through 24 and this is how range works.
Range is non-inclusive, which means that we ask for a range of 0 through 25. What we get 0 through 24, because it's up to but not including the range we specify. Range takes three possible arguments. It only requires 1 and so those three possible arguments are the start and the stop and the step. So we start at 0 and stop at 25 and step by 1, this is the result that we get.
And so start and step default to 0 and 1 and all you have to specify is the stop. If I were to start it say at 5 and step by say 2, save that and run it, you will get 5 through 23. Again, we don't get 25 because it's not inclusive, and you see they are stepping by 2. So what I would like to do here is create our own range object as an exercise to learn how to create a generator object in Python.
It will work exactly like this one, except that it will be inclusive. We'll always get that last number there. So go ahead and reset this to 25 and we will start creating our class. So we'll call this class inclusive_ range and it will have a constructor. And it will have an iterator. These are both special method names in Python. Init with the two underscores before and after it are for constructors and iter like this, with two underscores before it and two underscores after it, makes the object an iterable object.
So this is where we'll put our generator function in here. The constructor will need to be able to do this weird thing with the argument. The first argument and the third argument are optional but the second argument is required. That's going to take a little bit of manipulation on our end and we'll do that in the constructor. We need this arbitrary list of positional arguments. So we'll use the list argument syntax here and then asterisk and args and we'll get the number of arguments by using the length built in.
len(args) like that and if we have less than one argument we are going to raise an error. We'll use TypeError requires at least one argument. Let's say if numargs is less than 1 and we'll use elif numargs == 1, elif numargs == 2.
I like to outline my code like this sometimes and then go back and fill it in. Numargs = 3 or else we'll raise a TypeError. Expected at most three arguments and got this other number. Now we can go ahead and fill in each of these conditions. If we get just one argument then we know that's the stop. So we can say self.stop = args sub 0. Otherwise we have got self.start = 0 and self.step = 1, because those are those default values.
If we have two arguments and we know that that's the start and the stop. self.start, self.stop, and we assign that to args and we use our default for step. Finally, if we have three arguments, then we know that's all of them, start, stop and step. So now we have our constructor and our iterator is very easy.
We start by setting the starting point and we have a simple while loop, while i is less than or equal to a stop, then we yield the result-- and we'll get back to that one-- and we increment the iterator by this step. So this is what makes it a generator is the yield statement.
Yield works just like return but with the significant difference. If I were to use return here, it would return the value and the next time the iterator was called it would start at the beginning of the function. By using yield instead it returns the value and the next time the function is called execution picks up right after the yield statement. So the way this will run is it will set the starting point and it will test the while loop and assuming that the starting point is less than or equal to the stop point, it will yield the value. Do that's the starting point and then the next time the iterator is called, it will increment that value and test the while loop again, and then it will yield the next value and then it will increment and we'll yield the next value and this allows it to operate as an iterator.
So by having the yield statement inside the function, that makes the function a generator and what a generator generates is an iterable object. We need to change this to our inclusive_range. I have a little typo here. self.step. Save that and run it and there we have a range all the way up to and including our stop, and so let's go ahead and test out our constructor.
I'll change the start point to 5 and now it start to 5 and go up to 25. Save that and run it, and there it, starts at 5 and goes all the way up to 5. Let's have it step by 2 instead of stepping by 1. Save that and run it. Now we see it steps by 2. We can change that to 7 and it will still do what we expect. That's great and now let's call it with no arguments and we'll test our error conditions.
No arguments, we would expect it to get that TypeError and there it is, TypeError requires at least one argument and if we give it four arguments, let's say 1, 25, and 3 and 14, we expect to get this other type error here and there it is, expected at most three arguments and got four. So we have successfully duplicated the range generator function, save that and run it, except that ours is inclusive.
That was really easy to do. So this is how you make a generator object is by using the iter method in your class and now your object becomes iterable. It becomes a generator and when the object is used in the context of an iterable, like for instance, in this for loop, then that iter methods gets called automatically and you do not have to do something like .iter.
You can simply use the object in that context. In fact I don't even have to create an intermediate variable. So I can take this line out all together and put it in this place here and you will see that range is used like this very often. So I can save that and run it and there we have a drop in replacement for range except that this one is inclusive and that's how you make the generator object in Python. It's a very convenient and very powerful technique.
You will find yourself using it in some places that you won't anticipate. I have used it in database applications where I have a very specific database application with a specific class that simply steps through a particular type of a query. I have used it in parsing files, where I want to get a certain pattern out of a file. There is all kinds of applications for this and you will find that you actually use it more often than you might think.
- A Python 3 quick start for experienced developers
- Creating functions and objects
- Using Python's built-in objects and classes
- Repeating code with loops and iterators
- Understanding and using conditional expressions
- Creating sequences with generators
- Reusing code with objects and libraries
- Handling errors with exceptions