👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!
I have a script that"s successfully doing a multiprocessing Pool set of tasks with a imap_unordered()
call:
p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
p.join() # Wait for completion
However, my num_tasks
is around 250,000, and so the join()
locks the main thread for 10 seconds or so, and I"d like to be able to echo out to the command line incrementally to show the main process isn"t locked. Something like:
p = multiprocessing.Pool()
rs = p.imap_unordered(do_work, xrange(num_tasks))
p.close() # No more work
while (True):
remaining = rs.tasks_remaining() # How many of the map call haven"t been done yet?
if (remaining == 0): break # Jump out of while loop
print("Waiting for", remaining, "tasks to complete...")
time.sleep(2)
Is there a method for the result object or the pool itself that indicates the number of tasks remaining? I tried using a multiprocessing.Value
object as a counter (do_work
calls a counter.value += 1
action after doing its task), but the counter only gets to ~85% of the total value before stopping incrementing.
👻 Read also: what is the best laptop for engineering students?
We hope this article has helped you to resolve the problem. Apart from Show the progress of a Python multiprocessing pool imap_unordered call?, check other __main__ Python module-related topics.
Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.
By the way, this material is also available in other languages:
- Italiano Show the progress of a Python multiprocessing pool imap_unordered call?
- Deutsch Show the progress of a Python multiprocessing pool imap_unordered call?
- Français Show the progress of a Python multiprocessing pool imap_unordered call?
- Español Show the progress of a Python multiprocessing pool imap_unordered call?
- Türk Show the progress of a Python multiprocessing pool imap_unordered call?
- Русский Show the progress of a Python multiprocessing pool imap_unordered call?
- Português Show the progress of a Python multiprocessing pool imap_unordered call?
- Polski Show the progress of a Python multiprocessing pool imap_unordered call?
- Nederlandse Show the progress of a Python multiprocessing pool imap_unordered call?
- 中文 Show the progress of a Python multiprocessing pool imap_unordered call?
- 한국어 Show the progress of a Python multiprocessing pool imap_unordered call?
- 日本語 Show the progress of a Python multiprocessing pool imap_unordered call?
- हिन्दी Show the progress of a Python multiprocessing pool imap_unordered call?
Berlin | 2023-03-25
IMAP PHP module is always a bit confusing 😭 Show the progress of a Python multiprocessing pool imap_unordered call? is not the only problem I encountered. I just hope that will not emerge anymore
Abu Dhabi | 2023-03-25
I was preparing for my coding interview, thanks for clarifying this - Show the progress of a Python multiprocessing pool imap_unordered call? in Python is not the simplest one. Will use it in my bachelor thesis
Munchen | 2023-03-25
Maybe there are another answers? What Show the progress of a Python multiprocessing pool imap_unordered call? exactly means?. I just hope that will not emerge anymore