# How to use pmap with DrWatson

**URL:** <https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976>\
**Category:** New to Julia\
**Tags:** question\
**Created:** [May 28, 2021, 7:05am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976 "2021-05-28T07:05:41Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [May 28, 2021, 7:05am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/1 "2021-05-28T07:05:41Z")

</div>

I am using the great package `DrWatson` and I am trying to work with `pmap`.  
My project is also a package, and I am trying to use `@everywhere` to let the workers know about the function I am trying to run.  
The problem is that the function is not recognized. This is the error I get

```julia
ERROR: LoadError: UndefVarError: my_func not defined

```

The way I start the script is

```julia
using Distributed

@everywhere using DrWatson
@everywhere import Pkg
@everywhere Pkg.activate("path_to_project")
@everywhere using MyProject
addprocs(4)

```

Then by running

```julia
A = pmap(MyProject.my_func(), 1:10)

```

I get the error above.

How should I set up this script to work with `pmap` properly?

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [May 30, 2021, 8:43am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/2 "2021-05-30T08:43:29Z")

</div>

Hi @roi.holtzman,

try moving the `addprocs` command to the top.  
If they are only added AFTER calling `@everywhere using ...`  
then they will never know about it.

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [May 30, 2021, 9:25am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/3 "2021-05-30T09:25:14Z")

</div>

Thanks for the suggestion.  
I moved it, but it doesn’t help. I get the same error.

The code is

```julia
using Distributed
addprocs(4)
@everywhere using DrWatson
@everywhere import Pkg
@everywhere Pkg.activate("path_to_project")
@everywhere using MyProject

```

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [May 30, 2021, 9:48am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/4 "2021-05-30T09:48:06Z")

</div>

On my system, I put my projects in `.julia/dev` and enter `] dev MyProject` into the REPL. Using that setup, the following code works:

```julia
using Distributed
addprocs(4)
@everywhere begin 
    using MyProject
    # other dependencies
end

```

Note that you can enclose all of your dependencies within `@everywhere begin`.

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [May 30, 2021, 11:39am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/5 "2021-05-30T11:39:17Z")

</div>

In that case, I don’t think the distributed stuff is the problem.

Please verify, that your function is indeed defined with that name. (e.g. use `map` for testing or call it directly)

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [May 30, 2021, 2:11pm UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/6 "2021-05-30T14:11:06Z")

</div>

I am a bit confused.  
The function is defined well (I think).  
When I run

```julia
for i=1:N 
  my_func(a=1, b=2)
end

```

It runs fine.  
But when I run

```julia
map(my_func(a=1, b=2), 1:N)

```

It says that the function is not defined.  
So even `map` is not working well for me…  
What might be the problem?

---

<div class="post-metadata">

**Author:** ![Christopher\_Fisher](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/christopher_fisher/32/26132_2.png) [@Christopher\_Fisher](https://discourse.julialang.org/u/Christopher_Fisher)\
**Post date:** [May 30, 2021, 3:20pm UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/7 "2021-05-30T15:20:48Z")

</div>

I would not expect `my_func` to work with map, unless you have other methods defined. The problem is that you appear to be passing elements from 1:N to `my_func`, but there is no such argument to accept those elements. What I am finding is a method error:

```julia
julia> my_func(;a, b) = a + b
my_func (generic function with 1 method)

julia> map(my_func(a=1,b=2), 1:10)
ERROR: MethodError: objects of type Int64 are not callable
Stacktrace:
 [1] iterate
   @ ./generator.jl:47 [inlined]
 [2] _collect
   @ ./array.jl:691 [inlined]
 [3] collect_similar
   @ ./array.jl:606 [inlined]
 [4] map(f::Int64, A::UnitRange{Int64})
   @ Base ./abstractarray.jl:2294
 [5] top-level scope
   @ REPL[2]:1

```

Are you doing something different than that?

---

<div class="post-metadata">

**Author:** ![affans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/affans/32/11911_2.png) [@affans](https://discourse.julialang.org/u/affans)\
**Post date:** [May 30, 2021, 3:29pm UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/8 "2021-05-30T15:29:17Z")

</div>

Could you show the function signature for `my_func`?

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [May 30, 2021, 3:54pm UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/9 "2021-05-30T15:54:10Z")

</div>

Yes, I think there may be a misunderstanding what `map` does.

`map` takes a `function`.

Here is an example

```julia
map( x -> x^2, 0:4) 

```

gives  
`[0, 1, 4, 9, 16]`

so you have to iterate over the arguments!

If you need multiple arguments, you do

```julia
map(*, 1:2, 3:4) #== [1*3, 2*4]

```

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [May 30, 2021, 6:07pm UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/10 "2021-05-30T18:07:07Z")

</div>

I see! thanks for the explanation. It is indeed the first time I am using `map` and `pmap`.  
I have defined a wrapper function

```julia
@everywhere function pmap_wrap(dict)
    @unpack L, βinit, βfinal, h, J, Nequilibration, Niter, nskips = dict
	Ising2d.ising_single_realziation(L=L, βinit=βinit, βfinal=βfinal, h=h, J=J, Nequilibrate=Nequilibration, Niter=Niter, nskips=nskips, calc_correlation=true)
end

```

and now I am calling

```julia
dicts = fill(params, Nreal)
results_vector = map(pmap_wrap, dicts)

```

But I still have an error

```julia
ERROR: LoadError: UndefVarError: ising_single_realziation not defined

```

I am not sure what the problem is now… Even `map` is not working and I don’t see why.

---

<div class="post-metadata">

**Author:** ![JonasIsensee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jonasisensee/32/4704_2.png) [@JonasIsensee](https://discourse.julialang.org/u/JonasIsensee)\
**Post date:** [May 31, 2021, 6:40am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/11 "2021-05-31T06:40:29Z")

</div>

It is difficult to help you, when you never show all the code necessary to reproduce the problem.  
Please have a look at:

> [@Please read: make it easier to help you](https://discourse.julialang.org/t/psa-make-it-easier-to-help-you/14757):
>
> Welcome to the Julia Discourse! We are enthusiastic about helping Julia programmers, both beginner and experienced. This public service announcement (PSA) outlines best practices when asking for help. Following these points makes it easier for us to help you and more likely you’ll get a prompt, useful answer. Keywords are highlighted to make it easier to refer to specific points. Choose a descriptive title that captures the key part of your question, eg “plots with multiple axes” instead of …

One last guess : There is a typo in `ising_single_realziation` !?

---

<div class="post-metadata">

**Author:** ![roi.holtzman](https://avatars.discourse-cdn.com/v4/letter/r/f05b48/32.png) [@roi.holtzman](https://discourse.julialang.org/u/roi.holtzman)\
**Post date:** [May 31, 2021, 9:29am UTC](https://discourse.julialang.org/t/how-to-use-pmap-with-drwatson/61976/12 "2021-05-31T09:29:15Z")

</div>

You’re absolutely right!  
I tried to make my question clean, and I guess it only harmed it this time.  
I will try to do it better next time.

Anyways, I just had a typo, and now everything works well! Thank you so much for the help.

Just for the completeness of the discussion:  
To operate with `DrWatson` one needs to do the following:

```julia
using Distributed
addprocs(2)
@everywhere using DrWatson
@everywhere @quickactivate :Project

# do other things

```

Namely, `@quickactivate` works well with `Distributed`.

Note that using a `begin` block does _not_ work (the `@quickactivate` does not work well)

```julia
# this does not work!
@everywhere begin
    using DrWatson
    @quickactivate :Project
end

```