multiprocessing: sharing a large read-only object between processes?

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

Do child processes spawned via multiprocessing share objects created earlier in the program?

I have the following setup:

do_some_processing(filename):
    for line in file(filename):
        if line.split(",")[0] in big_lookup_object:
            # something here

if __name__ == "__main__":
    big_lookup_object = marshal.load("file.bin")
    pool = Pool(processes=4)
    print pool.map(do_some_processing, glob.glob("*.data"))

I"m loading some big object into memory, then creating a pool of workers that need to make use of that big object. The big object is accessed read-only, I don"t need to pass modifications of it between processes.

My question is: is the big object loaded into shared memory, as it would be if I spawned a process in unix/c, or does each process load its own copy of the big object?

Update: to clarify further - big_lookup_object is a shared lookup object. I don"t need to split that up and process it separately. I need to keep a single copy of it. The work that I need to split it is reading lots of other large files and looking up the items in those large files against the lookup object.

Further update: database is a fine solution, memcached might be a better solution, and file on disk (shelve or dbm) might be even better. In this question I was particularly interested in an in memory solution. For the final solution I"ll be using hadoop, but I wanted to see if I can have a local in-memory version as well.

👻 Read also: what is the best laptop for engineering students?

We hope this article has helped you to resolve the problem. Apart from multiprocessing: sharing a large read-only object between processes?, check other __main__ Python module-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

• Italiano multiprocessing: sharing a large read-only object between processes?
• Deutsch multiprocessing: sharing a large read-only object between processes?
• Français multiprocessing: sharing a large read-only object between processes?
• Español multiprocessing: sharing a large read-only object between processes?
• Türk multiprocessing: sharing a large read-only object between processes?
• Русский multiprocessing: sharing a large read-only object between processes?
• Português multiprocessing: sharing a large read-only object between processes?
• Polski multiprocessing: sharing a large read-only object between processes?
• Nederlandse multiprocessing: sharing a large read-only object between processes?
• 中文 multiprocessing: sharing a large read-only object between processes?
• 한국어 multiprocessing: sharing a large read-only object between processes?
• 日本語 multiprocessing: sharing a large read-only object between processes?
• हिन्दी multiprocessing: sharing a large read-only object between processes?