Change language

Downloading files from the Internet using Python

| | |

Requests — it is a versatile python HTTP library with various applications. One of its applications is to download a file from the Internet using the URL of the file. 
Installation: First of all, you need to download the requests library. You can install it directly using pip by entering the following command:

 pip install requests 

Or download it directly from here and install manually.

Downloading files

# imported the requests library

import requests

image_url = " -v3-TM.png "

# Image URL for upload is defined as image_url

r = requests.get (image_url) # create an HTTP response object

# send an HTTP request to the server and save
# HTTP response in response object named r

with open ( "python_logo.png" , ’ wb’ ) as f:


  # Save the resulting content as a png file

# binary


# write response content (r.content)

  # to a new file in binary mode .

f.write (r.content)

This little piece of code written above will download the following image from the Internet. Now check your local directory (the folder where this script is located) and you will find this image:

All we need is the URL of the image source. (You can get the URL of the image source by right-clicking on the image and selecting the View Image option.)

Download Large Files

HTTP response content ( r.content ) — it is nothing more than a string that file data is stored in. Thus, it will not be possible to keep all data in one line in case of large files. To overcome this problem, we are making some changes to our program:

  • Since all file data cannot be stored in one line, we use the r.iter_content method to load data into chunks specifying the chunk size.
  •  r = requests.get (URL, stream = True) 

    Setting the stream parameter to True will only download the response headers and the connection will remain open. This avoids concurrent reading of content into memory for large responses. The fixed chunk will be loaded every time the iteration of r.iter_content is running.

    Here’s an example:

    import requests

    file_url = " "


    r = requests.get (file_url, stream = True )


    with open ( < code class = "string"> "python.pdf" , "wb" ) as pdf:

      for chunk in r.iter_content (chunk_size = 1024 ):


    # write one piece at a time to a pdf

    if chunk:

      pdf.write (chunk)

    Video download

    In this example, we are interested in downloading all the video lectures available on this ve b-page . All archives for this lecture are available here . So, first we clean up the web page to extract all the links to the video, and then we load the video one at a time.

    import requests

    from bs4 import BeautifulSoup

    "" "
    URL of the archive web page with a link to
    all video lectures. It would be tedious
    download each video manually.
    In this example, we first scan a web page to extract
    all links and then download the video.
    "" "

    # enter the archive address here

    archive_url = " ~ csev / books / py4inf / media /"


    def get_video_links ():


    # create object response

    r = requests.get (archive_url)


    # create a nice soup object

    soup = BeautifulSoup (r.content, ’html5lib’ )


    # find all links on a web page

    links = soup.findAll ( ’a’ )


    # filter link submissions from using .mp4

    video_links = [archive_url + link [ ’ href’ ] for link in links if link [ ’href’ ]. endswith ( ’ mp4’ )]


      return video_links



    def download_video_series (video_links):

    < p>  

    for link in video_links:


      & # 39; & # 39; & # 39; iterate over all links in video_links

    and download them one by one


    # get the filename by splitting the URL and getting

    # last line

    file_name = link.split ( ’/’ ) [ - 1


    print "Downloading file:% s" % file_name


    # create response object

    r = requests.get (link, stream = True )


    # download started

    with open (file_name, ’wb’ ) as f:

      for chunk in r.iter_content (chunk_size = 1024 * 1024 ):

    if chunk:

    f.write (chunk)


    print "% s downloaded! " % file_name


    print "All videos downloaded!"




    if __ name__ = = " __ main__ " :


    # get all video links

    video_links = get_video_links ()


    # download all videos

    download_video_series (video_links)


    Benefits of using a query library to load web files:

    • You can easily download web directories by recursively browsing the website!
    • This is a browser independent method and much faster!
    • You can just flush the webpage to get all the file urls on the webpage, and hence load all files in one command.


      Learn programming in R: courses


      Best Python online courses for 2022


      Best laptop for Fortnite


      Best laptop for Excel


      Best laptop for Solidworks


      Best laptop for Roblox


      Best computer for crypto mining


      Best laptop for Sims 4


      Latest questions


      Common xlabel/ylabel for matplotlib subplots

      12 answers


      How to specify multiple return types using type-hints

      12 answers


      Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

      12 answers


      Flake8: Ignore specific warning for entire file

      12 answers


      glob exclude pattern

      12 answers


      How to avoid HTTP error 429 (Too Many Requests) python

      12 answers


      Python CSV error: line contains NULL byte

      12 answers


      csv.Error: iterator should return strings, not bytes

      12 answers


    Python | How to copy data from one Excel sheet to another

    Common xlabel/ylabel for matplotlib subplots

    Check if one list is a subset of another in Python


    How to specify multiple return types using type-hints


    Printing words vertically in Python


    Python Extract words from a given string

    Cyclic redundancy check in Python

    Finding mean, median, mode in Python without libraries


    Python add suffix / add prefix to strings in a list

    Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

    Python - Move item to the end of the list

    Python - Print list vertically