Show the progress of a Python multiprocessing pool imap_unordered call?

Michael Zippo

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

I have a script that"s successfully doing a multiprocessing Pool set of tasks with a imap_unordered() call:

p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
p.join() # Wait for completion

However, my num_tasks is around 250,000, and so the join() locks the main thread for 10 seconds or so, and I"d like to be able to echo out to the command line incrementally to show the main process isn"t locked. Something like:

p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
while (True):
  remaining = rs.tasks_remaining() # How many of the map call haven"t been done yet?
  if (remaining == 0): break # Jump out of while loop
  print("Waiting for", remaining, "tasks to complete...")
  time.sleep(2)

Is there a method for the result object or the pool itself that indicates the number of tasks remaining? I tried using a multiprocessing.Value object as a counter (do_work calls a counter.value += 1 action after doing its task), but the counter only gets to ~85% of the total value before stopping incrementing.

👻 Read also: what is the best laptop for engineering students?

We hope this article has helped you to resolve the problem. Apart from Show the progress of a Python multiprocessing pool imap_unordered call?, check other __main__ Python module-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

Davies Zelotti

Berlin | 2023-03-25

IMAP PHP module is always a bit confusing 😭 Show the progress of a Python multiprocessing pool imap_unordered call? is not the only problem I encountered. I just hope that will not emerge anymore

Boris Porretti

Abu Dhabi | 2023-03-25

I was preparing for my coding interview, thanks for clarifying this - Show the progress of a Python multiprocessing pool imap_unordered call? in Python is not the simplest one. Will use it in my bachelor thesis

Jan Emmerson

Munchen | 2023-03-25

Maybe there are another answers? What Show the progress of a Python multiprocessing pool imap_unordered call? exactly means?. I just hope that will not emerge anymore