Easy way to send custom function to distributed workers?

#1

Is there a way to pass a function to workers? e.g.

using Distributed
addprocs(4)

function my_func(x)
    return x ^ 2
end

[send my function to everyone]

pmap(my_func, 1:100)

I recognize I can do something like:

@everywhere begin
    function my_func(x)
        return x ^ 2
    end
end

But since functions are first class objects (right?) seems like I should just be able to pass them around like data.

I tried using ParallelDataTransfer.jl, as suggested here, but like @gdkrmr I can’t figure it out and the package has no docs.

For example, I tried:

@passobj 1 workers() my_func

But that didn’t seem to work…

0 Likes

#2

I think the issue here is that you want define methods (not just the function) on the workers.

I’m not sure how robust this is but it seems to work for your simple example:

function my_func(x)
    #println("Calculating my_func($x)")
    return x ^ 2
end

using Distributed
addprocs(4)

@everywhere workers() @eval function my_func end

for m in methods(my_func)
    @everywhere workers() @eval $m
end

pmap(my_func, 1:100)
1 Like

How to distribute the definitions of preexisting functions to workers?
#3

ahhhhh, right… methods / functions. I forgot about that distinction.

0 Likes