Sharing a result queue among several processes

Michael Zippo

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

The documentation for the multiprocessing module shows how to pass a queue to a process started with multiprocessing.Process. But how can I share a queue with asynchronous worker processes started with apply_async? I don"t need dynamic joining or anything else, just a way for the workers to (repeatedly) report their results back to base.

import multiprocessing
def worker(name, que):
    que.put("%d is done" % name)

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes=3)
    q = multiprocessing.Queue()
    workers = pool.apply_async(worker, (33, q))

This fails with: RuntimeError: Queue objects should only be shared between processes through inheritance. I understand what this means, and I understand the advice to inherit rather than require pickling/unpickling (and all the special Windows restrictions). But how do I pass the queue in a way that works? I can"t find an example, and I"ve tried several alternatives that failed in various ways. Help please?

👻 Read also: what is the best laptop for engineering students?

We hope this article has helped you to resolve the problem. Apart from Sharing a result queue among several processes, check other __main__ Python module-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

Schneider Lehnman

Tallinn | 2023-03-26

Thanks for explaining! I was stuck with Sharing a result queue among several processes for some hours, finally got it done 🤗. Checked yesterday, it works!

Marie Danburry

Paris | 2023-03-26

Thanks for explaining! I was stuck with Sharing a result queue among several processes for some hours, finally got it done 🤗. I am just not quite sure it is the best method

Walter Zelotti

Singapore | 2023-03-26

I was preparing for my coding interview, thanks for clarifying this - Sharing a result queue among several processes in Python is not the simplest one. Checked yesterday, it works!