Change language

Pythons Magical Itertools Module

Pythons Magical Itertools Module

[Music] in this video im going to be introducing you to an awesome module in python called itter tools now this module is indeed magical and the reason for that is it provides a bunch of built-in functionality specifically functions for creating iterators for efficient looping if im just quickly scrolling through here you can see we have stuff like count cycle repeat accumulate chain chain from iterable compress pairwise star map product permutations combinations and a ton of stuff that youve probably done by yourself from scratch simply because you did not know the function existed in a built-in module called itter tools so with that said lets head over to the code editor and ill start walking you through some of the more useful functions from the intertools module so the first thing im going to do here is give you a very quick explanation of what an iterator is because all of the intertools functions are going to return an iterator object and we should probably understand how that works now i will mention that if you do have my programming course programming expert then you would already understand iterators youd understand generators decorators more advanced programming object-oriented programming and all kinds of cool features in python so if youre interested check it out from the link in the description programming expert dot io you can use discount code tim anyways what is an iterator well to demonstrate this to you lets start by having a look at something known as the range function now weve probably seen this before but if i do something like r is equal to range and then lets go with a range of say 1 to 10 when i print this out here lets have a look and notice i simply get range 110 now range itself is not an iterator but its known as iterable now anything thats iterable simply has an iter method on so iter is this function that you can call and what this does is actually call the corresponding underscore underscore either which is on the class that represents this range function so i know this is a bit complicated but on range there is this underscore underscore either function that you can call when you call that it returns to you an iterator and the iterator is what you iterate over to get all of the values in the range created by this object so for example if i print out the iter of r youre going to see that i get a range iterator object now this range iterator object is something i can iterate over so lets have a look here lets just say i is equal to that and now lets print out for example the list of i when i do this i get one two three four five six seven eight nine because this is iterable so i can use the list function on it and it gives me all of the elements in this iterator however an iterator is an iterator when it has an underscore underscore next underscore underscore method so this method is a special method that returns to the next value in the iterable sequence so when i have a range of 1 to 10 what i can do is the following i can print out the next of my iterator which is i and lets print this out a few times now when i do this notice i get 1 2 and 3 right its giving me all of the values in my sequence and then to go even further lets have a look at something like this 4x in and then im going to say i because i can loop through an iterator right and i have my iterator i and when i start printing x here lets have a look and notice that i dont get any repeated values so i have my one two three from all of the necks up here and then when i continue looping through the iterator i get four five six seven eight nine because since weve called the next method manually weve been moved to the next values in the iterable sequence i know this is a little bit confusing if this is the first time that youre seeing it but essentially when you have an iterator or something thats iterable sorry you can call this iter function on it that returns to you an iterator and then the iterator you can call this next function on and it gives you the next value in the iterator sequence you know that youre finished iterating when a specific exception is raised this exception is known as stop iteration and to show you that lets just print next a bunch of times so we only have nine values in our sequence here so when i print the next of i more than nine times youll see here that we get a stop iteration exception and that tells us theres no more values in our iterator to be returned so that was a very high level overview of iterators and iterable objects however you may be interested to know that when we have a for loop like this and were looping through an iterable object what actually happens implicitly is youre going to call the iter function or method on this iterable object thats going to return an iterator and then the for loop is going to continually call the next method on this iterator until eventually the stop iteration exception is raised so once that exception is raised the for loop knows theres no more objects to loop through and it can stop iterating and the value that youre going to look at for i here for our iterator variable is going to be equal to whatever that next method returned at every single step in the iterator hopefully that makes a little bit of sense but again we have the next function we have the inner function and these are really just mapping to the dot underscore underscore next like this and to the dot and then this is going to be underscore underscore iter underscore underscore method so anything that is iterable has this iter method and anything that is an iterator has this next method and when you use the next function and the iter function youre just calling these methods on those objects all right hopefully that is clear now lets have a look at the itertools module so the first type of functions im going to show you from the editor tools module are known as infinite iterators and what that means is that they can return an infinite sequence and they actually do this by using something known as the generator syntax im not going to explain what a generator is in this video again you could learn that from something like programming expert but for now lets have a look at the first function which is count so this is an infinite iterator and the way this works is you pass a start and an optional step value if you dont pass a step this will by default be one and this will do exactly what it says it will simply count a sequence starting at some value and going up by the step every single time so if i start at 10 which is what im passing for the start here and then i have a step of five were going to count up by five until we decide we no longer want to iterate over this iterator so if i run the code here you can see that we get a sequence where we are starting at 10 going up by five and ive just made it so we manually stop here at a hundred if i didnt have this break then this would happen infinitely because again this is an infinite sequence so that is cool that is the count function now lets move on to the next one the next function that i have for you is also an infinite iterator and this is known as repeat now this is pretty straightforward but this is going to take in a element this element could be a list it could be a string it can be really anything you want and then the maximum number of times youd like to repeat this this is optional if you do not pass this it will repeat indefinitely so for here if i have repeat and im repeating hello and i say a maximum of 10 times then i can loop through this repeater iterator thats whats going to be returned here when you call the repeat function up to 10 times and print out the value so if i do this youll just see that we get 10 hellos showing up on the screen so this is useful if you want to repeat something a certain number of times so the next function that i have for you is known as cycle now this is pretty straightforward this is going to take in some iterable object could be a string could be a tuple could be a list really whatever you want and its going to allow you to cycle over all of the elements as many times as you would like again these are infinite iterators so you have to decide when you want to stop iterating over them they will continue to generate a sequence forever so lets run the code here and have a look at what this does notice that we just cycle through a b c d e f constantly until i have a manual stop here where weve done this over a hundred times so what im actually doing is manually calling the next method here on my cycler which is equal to a cycle iterator and if we want to just have a quick look here at what the iterator object actually looks like we can print out our cycler and notice here that i get an itertools.cycle object which is an iterator and we know its an iterator because it has this next method which i can manually call using the next function now i also could loop over this using a for loop if i wanted to so i could just do something like 4i in and then this would be cycler however if i did this i would need to implement a manual break condition because again that would go infinitely unless i manually broke it at some point so that is cycle again takes in some iterable and allows you to cycle over it as many times as you would like so the next set of iterators that im going to show you are known as terminating iterators theyre not infinite they do have a defined number of elements that they will return to you so the first iterator to look at here is known as accumulate and the best example of this is something like a running sum so i can say the running sum is equal to accumulate and then accumulate an iterable object in this case a bunch of integers and what this does is give me the sum of all of the elements at the current position and prior in the list so if i print this out here you see we get 1 3 6 10 so on and so forth so the first element is the sum of just the first element which is one second is the sum of these two elements the third is the sum of these three elements so on until you get to the very end and one thing to note here about this is that youve probably implemented this behavior by yourself in python before this will be much more memory efficient as well as faster than if you were just just to write this story with your own for loop and again the reason for this is its going to use the generator syntax its actually only going to give you these values when you request them its not going to be storing them in memory and taking up unnecessary space one other thing to note here is that whenever youre looking at an iterator so in this case we have accumulate we can call the list function on it directly and that will run the next method on the iterator until theres no more elements left and just collect all those values in a list so rather than manually looping through this using a for loop were calling the next method ourself were just using lists now to quickly grab the results of the iterate hopefully that makes sense that is accumulate so the next iterator to show you is known as chain this is very straightforward it simply chains two iterable objects together if i run this code you can see that we get abc and then def where the first iterable was this string and the second iterable was this one again this is going to be more performant and memory efficient than if you were to try to implement this behavior on your own and say concatenate two lists together this is not concatenating two lists it is returning to an iterator that allows you to retrieve one element at a time and process and use that as you need to that is the point of the iterators is that youre not storing everything in memory youre grabbing one individual element at a time and using it as you need to be as opposed to storing everything in memory when you only need the current element in the sequence hopefully that makes a bit of sense lets move on to the next example the next example is very similar to the first one but this is chain from iterable now lets just have a look at what this does when we pass in a nested list so notice here that i actually get this list flattened so this is something that you can do with the chain from interval pass in some type of nested structure it will actually flatten that structure for you by chaining all of the elements that are inside of this iterable object next well move on to compress as i was saying the next iterator that we have here is known as compress now what this does is take in some data source as well as some selectors and it simply keeps all of the items that are inside of this data if the corresponding item in the selector is true so in this case i have a nested structure that has three elements so this this and this then i pass in an array here i guess sorry a list that contains booleans now true indicates that were going to keep an element false means we are not going to keep it so if i run this youll see that we only keep a b and c because true was here for the first element now if i make this true youll see that now we get the second element as well and i could alternatively change these to be anything that returns a truthy value so i could put say zero and one and this would work as well moving on we have the pairwise function this one is cool its simply going to pair all of the adjacent elements in our iterable so lets run this and have a look and notice we get one two two three three four so on and so forth moving on the next set of iterators im going to show you are known as combinatronic iterators now the first one to have a look at here is a product now what the product is going to do is return the cartesian product of two iterable objects now this would be equivalent to kind of a nested for loop looping through every single possible pair of the items in iterable one and iterable two so lets have a look here when i print this out notice i get 1a 1b 1c 2a 2b 2c 3a 3b 3c that is what the cartesian product returns moving on we have the permutations iterator or function whatever youd like to refer to it as and what this does is return all of the permutations of a particular size of an iterable object so lets just have a look at what this returns we get a b a c a d b a b c so on and so forth and notice the order here is important so something like bc and cb are different permutations because the c and the b are in a different order that is not the same as what im going to show you next which is combinations so as i promised the next function to look at here is combinations and this is going to give you all of the combinations of a particular size of an iterable object so lets have a look at this and notice these are the combinations note there is a lot less combinations than there is permutations and the reason for that is that the ordering of the elements does not matter in a combination so ac and ca would be the same combination hence why were getting less pairs here now lets have a look at a size of three notice were going to have four possible combinations here if we have a look at four were only going to have one combination because theres only one unique combination of a b c and d so with that said i will start wrapping up the video here i will mention that these can become very powerful when you start combining them together so combining say the combinatronic iterator is with a terminating iterator maybe mixing in a map and filter function somewhere along the line and theres a bunch of other functions here that i did not show you from the intertools module so feel free to have a look at those from the documentation in the description hopefully you guys found some value from this video if you did make sure to leave a like subscribe to the channel and i will see you in another one [Music] you

Shop

Learn programming in R: courses

$

Best Python online courses for 2022

$

Best laptop for Fortnite

$

Best laptop for Excel

$

Best laptop for Solidworks

$

Best laptop for Roblox

$

Best computer for crypto mining

$

Best laptop for Sims 4

$

Latest questions

NUMPYNUMPY

Common xlabel/ylabel for matplotlib subplots

12 answers

NUMPYNUMPY

How to specify multiple return types using type-hints

12 answers

NUMPYNUMPY

Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

12 answers

NUMPYNUMPY

Flake8: Ignore specific warning for entire file

12 answers

NUMPYNUMPY

glob exclude pattern

12 answers

NUMPYNUMPY

How to avoid HTTP error 429 (Too Many Requests) python

12 answers

NUMPYNUMPY

Python CSV error: line contains NULL byte

12 answers

NUMPYNUMPY

csv.Error: iterator should return strings, not bytes

12 answers

News


Wiki

Python | How to copy data from one Excel sheet to another

Common xlabel/ylabel for matplotlib subplots

Check if one list is a subset of another in Python

sin

How to specify multiple return types using type-hints

exp

Printing words vertically in Python

exp

Python Extract words from a given string

Cyclic redundancy check in Python

Finding mean, median, mode in Python without libraries

cos

Python add suffix / add prefix to strings in a list

Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

Python - Move item to the end of the list

Python - Print list vertically