NLP | Parallel list processing with execnet

Michael Zippo 18.07.2021

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

In the code below, the integers are simply doubled, any clean computation can be done. This is the module to be executed by execnet. It receives 2 tuples (i, arg), assumes arg is a number, and sends back (i, arg * 2).

Code :

if __ name__ = = ’__channelexec__’ :

for (i, arg) in channel:

channel.send ((i, arg * 2 ))

To use this module to double each item in the list, import the plists module and call plists .map () with a remote_double module and a list of integers to double.

Code: Using plist

import plists, remote_double

plists. map (remote_double, range ( 10 ))

Output:

 [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

The map () function is defined in plists.py. It takes a pure module, an argument list, and an optional two-tuple list of (spec, count). By default, the specs are used [(& # 39; popen & # 39 ;, 2)], which means that the user will open two local gateways and channels. Once these pipes are open, the user can place them in an itertools loop, which creates an infinite iterator that goes back to the beginning as soon as it reaches the end.

Now each argument can be sent as arguments to the pipe for processing , and since the channels are cyclical, each channel receives an almost equal distribution of arguments. That’s where I came in — the order in which the results are returned is unknown, so i , as the index of each argument in the list, is passed to and from the channel so that the user can combine the results in the original order. Then wait for the results with the MultiChannel receive queue and insert them into a prepopulated list of the same length as the original arguments. After getting all expected results, exit the gateways and return the results as shown in the code below —

Code :

import itertools, execnet

def map (mod, args, specs = [( ’popen’ , 2 )]):

gateways = []

channels = []

for spec, count in specs:

for i in range (count):

gw = execnet.makegateway (spec)

gateways.append (gw)

channels.append (gw.remote_exec (mod))

cyc = itertools.cycle (channels)

for i, arg in enumerate (args):

channel = next (cyc )

channel.send ((i, arg))

mch = execnet.MultiChannel (channels)

queue = mch.make_receive_queue ()

l = len (args)

# creates a list of length l,

# where each element is None

results = [ None ] * l

for i in range (l):

channel, (i, result) = queue.get ()

results [i] = result

for gw in gateways:

gw.exit ()

return results

Code: Increase parallelization by changing the spec

plists. map (remote_double, range ( 10 ), [( ’popen’ , 4 )])

Output:

 [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

However, no more parallelization necessarily means faster processing. It depends on the resources available, and the more gateways and channels open, the more overhead. Ideally, there should be one gateway and channel per CPU core to maximize resource utilization. Use plists.map () with any clean module as long as it receives and sends back 2 tuples, where i is the first element. This pattern is most useful when there are many numbers that need to be processed in order to process them as quickly as possible.

👻 Read also: what is the best laptop for engineering students?

We hope this article has helped you to resolve the problem. Apart from NLP | Parallel list processing with execnet, check other ast Python module-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

Anna Emmerson

London | 2023-03-21

ast Python module is always a bit confusing 😭 NLP | Parallel list processing with execnet is not the only problem I encountered. Will use it in my bachelor thesis

Angelo Emmerson

California | 2023-03-21

Simply put and clear. Thank you for sharing. NLP | Parallel list processing with execnet and other issues with ast Python module was always my weak point 😁. I just hope that will not emerge anymore

Davies Jackson

California | 2023-03-21

Maybe there are another answers? What NLP | Parallel list processing with execnet exactly means?. I just hope that will not emerge anymore