Make functions in the present model available to workers

General Usage Performance
1

Hello.

I am trying to use pmap to call a function that is defined inside the present module, but I can’t make it work. I’ve searched for a few hours for the solution. Therefore I decided to write here. This is a minimal working example.

# ./src/hello.jl
module hello
export B

using Distributed

function A(x)
    return x 
end 

function B()
    addprocs(1)
    println( pmap(A, [1,2,3]) )
end

end # module hello

And

# ./test/test.jl
using hello
B()

This produces the error KeyError: key hello [61aee02f-779e-4bbf-b128-521c45f4bdad] not found

Can I solve this without changing the design too much? I want to give the workers access to the functions defined in the present module.

2

You need to load the definitions on the workers as well. I recommend checking out
https://docs.julialang.org/en/v1/manual/distributed-computing/#code-availability

3

Would you like to suggest any code?

4

Try maybe:

# ./test/test.jl
using Distributed
addprocs(1) # add the workers
# maybe you need to activate the current project on every worker
# or maybe you need to include the file hello.jl on every worker
# I don't know your setup as it is not included in your example
@everywhere using hello # load code everywhere
B()

and change B to

function B()
    println( pmap(A, [1,2,3]) )
end

Generally it is helpful to separate the setup that should only run once (like addprocs) from code that actually solves the problem.

5

Thanks for the suggestion. When I try that, I get the following error.


ERROR: LoadError: On worker 2:
ArgumentError: Package hello not found in current path.
- Run `import Pkg; Pkg.add("hello")` to install the hello package.

What do you think is causing this?

6

You need to activate the environment on the workers as well. So do something like:

@everywhere import Pkg
@everywhere Pkg.activate(".")

This should be explained in the documentation I linked, or is it too hard to understand? In that case please give some feedback so we can improve it.

7

Ok. The thing you said worked. I will try to rewrite my application to add processes from outside the module, as in the hello-project.

In case you want feedback on the documentation, as you say; the main issue for me was that I couldn’t get it to work for such a long time that I lost confidence that it was going to work. I was beginning to anticipate that the problem was with something other than the multi-processing. At that point I asked the forum because I didn’t have any drive left in me.

As with many topics in Julia, there aren’t any lectures that runs through the multiprocessing from beginning to end. I have the impression that the documentation relies, in my taste, too much on examples, which isn’t conceptual enough for me in many cases (such as in the present case).

1 Like