I researched first and couldn"t find an answer to my question. I am trying to run multiple functions in parallel in Python.
I have something like this:
files.py import common #common is a util class that handles all the IO stuff dir1 = "C:folder1" dir2 = "C:folder2" filename = "test.txt" addFiles = [25, 5, 15, 35, 45, 25, 5, 15, 35, 45] def func1(): c = common.Common() for i in range(len(addFiles)): c.createFiles(addFiles[i], filename, dir1) c.getFiles(dir1) time.sleep(10) c.removeFiles(addFiles[i], dir1) c.getFiles(dir1) def func2(): c = common.Common() for i in range(len(addFiles)): c.createFiles(addFiles[i], filename, dir2) c.getFiles(dir2) time.sleep(10) c.removeFiles(addFiles[i], dir2) c.getFiles(dir2)
I want to call func1 and func2 and have them run at the same time. The functions do not interact with each other or on the same object. Right now I have to wait for func1 to finish before func2 to start. How do I do something like below:
process.py from files import func1, func2 runBothFunc(func1(), func2())
I want to be able to create both directories pretty close to the same time because every min I am counting how many files are being created. If the directory isn"t there it will throw off my timing.
Due to peculiarities of CPython,
threading is unlikely to achieve true parallelism. For this reason,
multiprocessing is generally a better bet.
Here is a complete example:
from multiprocessing import Process def func1(): print "func1: starting" for i in xrange(10000000): pass print "func1: finishing" def func2(): print "func2: starting" for i in xrange(10000000): pass print "func2: finishing" if __name__ == "__main__": p1 = Process(target=func1) p1.start() p2 = Process(target=func2) p2.start() p1.join() p2.join()
The mechanics of starting/joining child processes can easily be encapsulated into a function along the lines of your
def runInParallel(*fns): proc =  for fn in fns: p = Process(target=fn) p.start() proc.append(p) for p in proc: p.join() runInParallel(func1, func2)
The rate at which we produce data is growing steadily, thus creating even larger streams of continuously evolving data. Online news, micro-blogs, search queries are just a few examples of these contin...
Python: - The Bible - 3 Manuscripts in 1 book:
90 SPECIFIC WAYS TO WRITE BETTER PYTHON. The Python programming language has unique strengths and charms that can be hard to grasp. Many programmers familiar with other languages often approach Pyt...
Data and storage models are the basis for big data ecosystem stacks. While storage model captures the physical aspects and features for data storage, data model captures the logical representation and...