Multiprocessing in Python | Kit 1 (Introduction)

This article — a short but short introduction to multiprocessing in the Python programming language.

What is multiprocessing?

Under multiprocessing refers to the ability of a system to support more than one processor at a time. Applications on a multiprocessor system are broken down into smaller routines that run independently. The operating system distributes these threads among the processors, improving system performance.

Why multiprocessing?

Consider a computer system with one processor. If it has multiple processes assigned at the same time, it will have to interrupt each task and briefly switch to another for all processes to work. 
This situation is similar to how a cook works alone in a kitchen. It has to do several tasks like baking, stirring, kneading dough, etc.

So the bottom line is: the more tasks you have to complete at the same time, the more difficult it becomes to keep track of them all, and keeping to the right time is becoming more and more challenging. 
This is where the concept of multiprocessing comes in! 
A multiprocessor system can have:

  • a multiprocessor, that is, a computer with more than one central processing unit.
  • a multi-core processor, that is one computational component with two or more independent actual processors (called “cores”).

Here a processor can easily execute multiple tasks at the same time, with each task using its own processor.

It`s like a chef in the last situation, assisted by his assistants. Now they can share tasks among themselves, and the chef does not need to switch between his tasks.

Multiprocessing in Python

In Python module multiprocessing includes a very simple and intuitive API for division of work among several processes. 
Let`s take a look at a simple example of using a multiprocessing module:

# multiprocessor module import

import multiprocessing


def print_cube (num):

"" "

  function for printing a cube with a given number

"" "

  print ( " Cube: {} " . format (num * num * num))


def print_square (num):

"" "

function for printing a square of a given number

  "" "

  print ( "Square: {}" . format (num * num))


if __ name__ = = "__ main__" :

# creating processes

  p1 = multiprocessing.Process (target = print_square, args = ( 10 ,))

p2 = multiprocessing.Process (target = print_cube, args = ( 10 ,))


# start process 1

p1.start ()

# start process 2

p2.start ()


# wait for process 1 to finish

p1.join ()

# wait for process to end 2

p2.join ()


# both processes completed

print ( "Done!" )

 Square: 100 Cube: 1000 Done! 

Let`s try to understand the above code:

  • To import a multiprocessing module, we do:
     import multiprocessing 
  • To create process, we create an object of class Process . The following arguments are required:
    • target : function to be executed by the process
    • args : arguments to pass to the target function
    • Note. The process constructor also accepts many other arguments, which will be discussed later. In the above example, we have created 2 processes with different target functions:

       p1 = multiprocessing.Process (target = print_square, args = (10,)) p2 = multiprocessing.Process (target = print_cube, args = ( 10,)) 
    • To start the process, we use the start method of the Process class.
       p1.start () p2. start () 
    • After starting the processes, the current program also continues to run. To stop the execution of the current program before the process ends, we use the join method.
       p1.join () p2.join () 

      As a result, the current program will first wait for completion p1, and then p2 . Once they are complete, the following statements of the current program are executed.

    Let`s take a look at another program to understand the concept of different processes running on the same Python script. In this example below, we print the identifier of the processes that perform the target functions:

    # multiprocessor module import

    import multiprocessing

    import os


    def worker1 ():

    # print process ID

    print ( "ID of process running worker1: {}" . format (os.getpid ()))


    def worker2 ():

    # ID of the printing process

      print ( " ID of process running worker2: {} " . format (os.getpid ()))


    if __ name__ = = "__ main__" :

    # print the process ID of the main program

    print ( "ID of main process: {}" . format (os.getpid ()))


    # create processes

    p1 = multiprocessing. Process (target = worker1)

    p2 = multiprocessing.Process (target = worker2)


    # starting processes

    p1.start ()

    p2.start ()


    # process identifiers

    print ( "ID of process p1: {}" . format (p1.pid))

    print ( "ID of process p2: {}" . format (p2.pid))


    # wait for processes to finish

      p1.join ()

    p2.join ()


    # both processes completed

    print ( "Both processes finished execution!" )


    # check if processes are alive

    print ( " Process p1 is alive: {} " . format (p1.is_alive ()))

    print ( " Process p2 is alive: {} " . format (p2.is_alive ()))

     ID of main process: 28628 ID of process running worker1: 29305 ID of process running worker2: 29306 ID of process p1: 29305 ID of process p2: 29306 Both processes finished execution! Process p1 is alive: False Process p2 is alive: False 
    • The main Python script has a different process ID and the multiprocessing module spawns new processes with different process IDs when we create process objects p1 and p2 . In the above program, we use the os.getpid () function, to get the process ID of the current target function.

      Note that it matches the process IDs of p1 and p2, which we get using the pid attribute of the Process class.

    • Each process runs independently and has its own memory space.
    • As soon as execution of the target function is finished, processes are terminated. In the above program, we used the is_alive method of the Process, class to check if the process is active or not.

    Consider the diagram below to understand how the new processes differ from the main Python script:

    So this was a short introduction to multiprocessing in Python ... The next few articles will focus on the following topics related to multiprocessing:

    • Interprocess communication using array, value, and queues.
    • Locking and pooling concepts in a multiprocessing environment


Get Solution for free from DataCamp guru