How to use pmap with DrWatson

New to Julia
1

I am using the great package DrWatson and I am trying to work with pmap.
My project is also a package, and I am trying to use @everywhere to let the workers know about the function I am trying to run.
The problem is that the function is not recognized. This is the error I get

ERROR: LoadError: UndefVarError: my_func not defined

The way I start the script is

using Distributed

@everywhere using DrWatson
@everywhere import Pkg
@everywhere Pkg.activate("path_to_project")
@everywhere using MyProject
addprocs(4)

Then by running

A = pmap(MyProject.my_func(), 1:10)

I get the error above.

How should I set up this script to work with pmap properly?

2

Hi @roi.holtzman,

try moving the addprocs command to the top.
If they are only added AFTER calling @everywhere using ...
then they will never know about it.

3

Thanks for the suggestion.
I moved it, but it doesn’t help. I get the same error.

The code is

using Distributed
addprocs(4)
@everywhere using DrWatson
@everywhere import Pkg
@everywhere Pkg.activate("path_to_project")
@everywhere using MyProject
4

On my system, I put my projects in .julia/dev and enter ] dev MyProject into the REPL. Using that setup, the following code works:

using Distributed
addprocs(4)
@everywhere begin 
    using MyProject
    # other dependencies
end

Note that you can enclose all of your dependencies within @everywhere begin.

2 Likes
5

In that case, I don’t think the distributed stuff is the problem.

Please verify, that your function is indeed defined with that name. (e.g. use map for testing or call it directly)

6

I am a bit confused.
The function is defined well (I think).
When I run

for i=1:N 
  my_func(a=1, b=2)
end

It runs fine.
But when I run

map(my_func(a=1, b=2), 1:N)

It says that the function is not defined.
So even map is not working well for me…
What might be the problem?

7

I would not expect my_func to work with map, unless you have other methods defined. The problem is that you appear to be passing elements from 1:N to my_func, but there is no such argument to accept those elements. What I am finding is a method error:

julia> my_func(;a, b) = a + b
my_func (generic function with 1 method)

julia> map(my_func(a=1,b=2), 1:10)
ERROR: MethodError: objects of type Int64 are not callable
Stacktrace:
 [1] iterate
   @ ./generator.jl:47 [inlined]
 [2] _collect
   @ ./array.jl:691 [inlined]
 [3] collect_similar
   @ ./array.jl:606 [inlined]
 [4] map(f::Int64, A::UnitRange{Int64})
   @ Base ./abstractarray.jl:2294
 [5] top-level scope
   @ REPL[2]:1

Are you doing something different than that?

8

Could you show the function signature for my_func?

9

Yes, I think there may be a misunderstanding what map does.

map takes a function.

Here is an example

map( x -> x^2, 0:4)

gives
[0, 1, 4, 9, 16]

so you have to iterate over the arguments!

If you need multiple arguments, you do

map(*, 1:2, 3:4) #== [1*3, 2*4]
10

I see! thanks for the explanation. It is indeed the first time I am using map and pmap.
I have defined a wrapper function

@everywhere function pmap_wrap(dict)
    @unpack L, βinit, βfinal, h, J, Nequilibration, Niter, nskips = dict
	Ising2d.ising_single_realziation(L=L, βinit=βinit, βfinal=βfinal, h=h, J=J, Nequilibrate=Nequilibration, Niter=Niter, nskips=nskips, calc_correlation=true)
end

and now I am calling

dicts = fill(params, Nreal)
results_vector = map(pmap_wrap, dicts)

But I still have an error

ERROR: LoadError: UndefVarError: ising_single_realziation not defined

I am not sure what the problem is now… Even map is not working and I don’t see why.

11

It is difficult to help you, when you never show all the code necessary to reproduce the problem.
Please have a look at:

One last guess : There is a typo in ising_single_realziation !?

1 Like
12

You’re absolutely right!
I tried to make my question clean, and I guess it only harmed it this time.
I will try to do it better next time.

Anyways, I just had a typo, and now everything works well! Thank you so much for the help.

Just for the completeness of the discussion:
To operate with DrWatson one needs to do the following:

using Distributed
addprocs(2)
@everywhere using DrWatson
@everywhere @quickactivate :Project

# do other things

Namely, @quickactivate works well with Distributed.

Note that using a begin block does not work (the @quickactivate does not work well)

# this does not work!
@everywhere begin
    using DrWatson
    @quickactivate :Project
end