Use numpy array in shared memory for multiprocessing

Michael Zippo

👻 Check our latest review to choose the best laptop for Machine Learning engineers and Deep learning tasks!

I would like to use a numpy array in shared memory for use with the multiprocessing module. The difficulty is using it like a numpy array, and not just as a ctypes array.

from multiprocessing import Process, Array
import scipy

def f(a):
    a[0] = -a[0]

if __name__ == "__main__":
    # Create the array
    N = int(10)
    unshared_arr = scipy.rand(N)
    arr = Array("d", unshared_arr)
    print "Originally, the first two elements of arr = %s"%(arr[:2])

    # Create, start, and finish the child processes
    p = Process(target=f, args=(arr,))
    p.start()
    p.join()

    # Printing out the changed values
    print "Now, the first two elements of arr = %s"%arr[:2]

This produces output such as:

Originally, the first two elements of arr = [0.3518653236697369, 0.517794725524976]
Now, the first two elements of arr = [-0.3518653236697369, 0.517794725524976]

The array can be accessed in a ctypes manner, e.g. arr[i] makes sense. However, it is not a numpy array, and I cannot perform operations such as -1*arr, or arr.sum(). I suppose a solution would be to convert the ctypes array into a numpy array. However (besides not being able to make this work), I don"t believe it would be shared anymore.

It seems there would be a standard solution to what has to be a common problem.

👻 Read also: what is the best laptop for engineering students?

We hope this article has helped you to resolve the problem. Apart from Use numpy array in shared memory for multiprocessing, check other __main__ Python module-related topics.

Want to excel in Python? See our review of the best Python online courses 2023. If you are interested in Data Science, check also how to learn programming in R.

By the way, this material is also available in other languages:

Jan Emmerson

Abu Dhabi | 2023-03-25

Thanks for explaining! I was stuck with Use numpy array in shared memory for multiprocessing for some hours, finally got it done 🤗. Will use it in my bachelor thesis

Boris OConnell

Warsaw | 2023-03-25

StackOverflow is always a bit confusing 😭 Use numpy array in shared memory for multiprocessing is not the only problem I encountered. I am just not quite sure it is the best method

Manuel Sikorski

Paris | 2023-03-25

multiprocessing Python module is always a bit confusing 😭 Use numpy array in shared memory for multiprocessing is not the only problem I encountered. Will get back tomorrow with feedback