How to run functions in parallel?

StackOverflow

I researched first and couldn"t find an answer to my question. I am trying to run multiple functions in parallel in Python.

I have something like this:

files.py

import common #common is a util class that handles all the IO stuff

dir1 = "C:folder1"
dir2 = "C:folder2"
filename = "test.txt"
addFiles = [25, 5, 15, 35, 45, 25, 5, 15, 35, 45]

def func1():
   c = common.Common()
   for i in range(len(addFiles)):
       c.createFiles(addFiles[i], filename, dir1)
       c.getFiles(dir1)
       time.sleep(10)
       c.removeFiles(addFiles[i], dir1)
       c.getFiles(dir1)

def func2():
   c = common.Common()
   for i in range(len(addFiles)):
       c.createFiles(addFiles[i], filename, dir2)
       c.getFiles(dir2)
       time.sleep(10)
       c.removeFiles(addFiles[i], dir2)
       c.getFiles(dir2)

I want to call func1 and func2 and have them run at the same time. The functions do not interact with each other or on the same object. Right now I have to wait for func1 to finish before func2 to start. How do I do something like below:

process.py

from files import func1, func2

runBothFunc(func1(), func2())

I want to be able to create both directories pretty close to the same time because every min I am counting how many files are being created. If the directory isn"t there it will throw off my timing.

Answer rating: 202

You could use threading or multiprocessing.

Due to peculiarities of CPython, threading is unlikely to achieve true parallelism. For this reason, multiprocessing is generally a better bet.

Here is a complete example:

from multiprocessing import Process

def func1():
  print "func1: starting"
  for i in xrange(10000000): pass
  print "func1: finishing"

def func2():
  print "func2: starting"
  for i in xrange(10000000): pass
  print "func2: finishing"

if __name__ == "__main__":
  p1 = Process(target=func1)
  p1.start()
  p2 = Process(target=func2)
  p2.start()
  p1.join()
  p2.join()

The mechanics of starting/joining child processes can easily be encapsulated into a function along the lines of your runBothFunc:

def runInParallel(*fns):
  proc = []
  for fn in fns:
    p = Process(target=fn)
    p.start()
    proc.append(p)
  for p in proc:
    p.join()

runInParallel(func1, func2)




Get Solution for free from DataCamp guru