Pmap in function/module

New to Julia
1

I’m having trouble running pmap when it’s inside a function in a module. Here is a MWE:

In myModule.jl

module myModule

    using Distributed

    function myOuterFunction()

        myVec = collect(1.0:0.1:10.0)

        @everywhere function f(x::Float64)
            return x*x
        end

        retVec = pmap(x->f(x), myVec)
        return retVec
    end
end

and in myscript.jl:

using Distributed
addprocs(8)

include("myModule.jl")

myModule.myOuterFunction()

yields an error:

ERROR: On worker 2:
UndefVarError: myModule not defined

If I instead do

@everywhere include("myModule.jl")

I get the error

ERROR: On worker 3:
UndefVarError: f not defined

even though the function definition is @everywhere’d.

Any help is much appreciated.

1 Like
2

It seems that the @everywhere definition of the function f also sends namespace information to the workers, and does not just define a function f in the global scope.

This works:

@everywhere module myModule

    using Distributed

    function f(x::Float64)
        return x*x
    end

    function myOuterFunction()

        myVec = collect(1.0:0.1:10.0)
            
        retVec = pmap(x->f(x), myVec)
        return retVec
    end
end
4 Likes
3

Thanks a lot. I guess this makes it hard (impossible?) to add workers within the function, do the pmap, and then destroy them again. Is it encouraged to start the workers at the very beginning and keep them throughout?

4

Yo can remove workers with rmprocs(), as stated in the docs. Nevertheless, you will have some overhead if you want to load the workers again.

About removing them after the pmap, I would leave them until the script finishes, I don’t think they will add some overhead.