Easy way to send custom function to distributed workers?

Specific Domains Julia at Scale
1

Is there a way to pass a function to workers? e.g.

using Distributed
addprocs(4)

function my_func(x)
    return x ^ 2
end

[send my function to everyone]

pmap(my_func, 1:100)

I recognize I can do something like:

@everywhere begin
    function my_func(x)
        return x ^ 2
    end
end

But since functions are first class objects (right?) seems like I should just be able to pass them around like data.

I tried using ParallelDataTransfer.jl, as suggested here, but like @gdkrmr I can’t figure it out and the package has no docs.

For example, I tried:

@passobj 1 workers() my_func

But that didn’t seem to work…

1 Like
2

I think the issue here is that you want define methods (not just the function) on the workers.

I’m not sure how robust this is but it seems to work for your simple example:

function my_func(x)
    #println("Calculating my_func($x)")
    return x ^ 2
end

using Distributed
addprocs(4)

@everywhere workers() @eval function my_func end

for m in methods(my_func)
    @everywhere workers() @eval $m
end

pmap(my_func, 1:100)
4 Likes
3

ahhhhh, right… methods / functions. I forgot about that distinction.

4

Is this the most up-to-date solution? I’ve spent hours trying to find a oneliner, not having to iterate over the methods.

But I’m grateful that this works, thanks!

1 Like