# UndefVarError when initializing variable inside function with @everywhere

**URL:** <https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183>\
**Category:** Julia at Scale\
**Created:** [June 12, 2019, 11:46am UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183 "2019-06-12T11:46:54Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 12, 2019, 11:46am UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/1 "2019-06-12T11:46:54Z")

</div>

I am trying to solve a problem in which I have to evaluate a function, f(x), for several different values of an argument x. I want to write a function that depending on the number of available processes it uses either map (if `nprocs()==1`) or pmap (if `nprocs()>1`). The function f(x) I want to evaluate requires a preallocated matrix and this is where I find a behaviour I do not understand.

This minimal example shows the problem I am facing:

```julia
using Distributed
addprocs(4)
@everywhere using LinearAlgebra

function test1()
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return eigvals(A)
    else
        B = zeros(100, 100)
        return eigvals(B)
    end
end

function test2()
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return eigvals(A)
    else
        A = zeros(100, 100)
        return eigvals(A)
    end
end

```

If I run `test1()`, it works without problems. However, when I run `test2()`, I get the error: `UndefVarError: A not defined`

If I run the code without adding any procs, both `test1()` and `test2()` work. Of course I can just write my code in the style of `test1()`, but I would like to understand why `test2()` fails. Thanks in advance for any suggestion!

EDIT: Changed the title of the topic to reflect the fact that the issue seems to be unrelated to the use of and `if` clause

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [June 12, 2019, 1:03pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/2 "2019-06-12T13:03:28Z")

</div>

In general this seems like a bug: the `if` parts of `if/else` clause of the two methods seem completely identical to me (unless I am missing a trick), so it is not clear why the `else` part that does not get executed would cause an error.

To recap what is going on here, however, with `addprocs` this code spawns four child processes, and the `@everywhere ... ` statement creates a global variable `A` both on parent and on all children. The calls to `eigval`, that are not prefixed by `@spawn` or any other macro from `Distributed`, will execute on master only. Is this really the intention?

---

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 12, 2019, 1:30pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/3 "2019-06-12T13:30:14Z")

</div>

To give some more context:  
In the actual problem I am trying to solve, I will run the `eigvals` function inside a map/pmap. So a slightly less minimal example would be:

```julia
using Distributed
addprocs(4)

@everywhere using Random, LinearAlgebra

# function which I want to run using map/pmap (it fills a preallocated matrix, mat, with random numbers and computes its eigenvalues) for different values of x.
@everywhere function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v1(iter)
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return pmap(x->h!(A, x), iter)
    else
        B = zeros(100, 100)
        return map(x->h!(B, x), iter)
    end
end

function run_h_v2(iter)
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return pmap(x->h!(A, x), iter)
    else
        A = zeros(100, 100)
        return map(x->h!(A, x), iter)
    end
end

```

So in order to avoid allocating a new matrix for each value of `x`, I preallocate a matrix `A`, which is modified by the function `h!(mat, x)`. When several processes are available, I initialize the `A` matrix in all processes using `@everywhere`.

Now, `run_h_v1(1:50)` runs fine, while `run_h_v2(1:50)` throws the same error as the example in the original post:

```julia
On worker 2:
UndefVarError: A not defined

```

As the example in the original post indicates, the error seems to caused by the use of `@everywhere` inside an `if`. Is this really a bug or I am missing something?

P.S.: I can probably just always use `pmap` for both cases (`nproc()==1` and `nproc()>1`) as it appears there is no overhead in using pmap instead of map. But I would still like to understand what is the origin of this issue.

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [June 12, 2019, 2:23pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/4 "2019-06-12T14:23:17Z")

</div>

Just pre-allocate `A` outside of the function. The `@everywhere` macro executes just the statement that is passed to it, so `A` will be in global scope on the child processes anyway.

The code below works, regardless of whether `addprocs` were called or not.

```julia
using Distributed
addprocs(4)
@everywhere using LinearAlgebra, Random

@everywhere A = zeros(100, 100)

@everywhere function h!(x)

    return eigvals(rand!(MersenneTwister(x), A))

end

function run_h(iter)
    if nprocs()>1
        return pmap(x->h!(x), iter)
    else
        return map(x->h!(x), iter)
    end
end

```

---

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 12, 2019, 5:04pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/5 "2019-06-12T17:04:03Z")

</div>

Thank you [tanhevg](https://discourse.julialang.org/t/declaration-of-variables-inside-function-with-everywhere-and-conditional-evaluation/25183/4) for your suggestion! Indeed that works, but I would like for the matrix `A` to be defined only when the function `run_h` is called (I want to include the code inside a module and the size of the matrix `A` might always not be the same).

But there is something about the use of `@everywhere` inside a function that I am probably missing and the error I am getting might be unrelated to the use of an `if` clause. For example the following code works fine in an interactive environment (IJulia):

```julia
using Distributed

addprocs(2)

@everywhere using LinearAlgebra, Random

@everywhere function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v3(iter)
    @everywhere A = zeros(100, 100)
    return pmap(x->h!(A, x), iter)
end

run_h_v3(1:20)

```

However, if I move the code to a module `run_h.jl`:

```julia
module run_h

using Distributed, Random, LinearAlgebra

export run_h_v3

function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v3(iter)
    @everywhere A = zeros(100, 100)
    return pmap(x->h!(A, x), iter)
end

end

```

and then do in an interactive environment

```julia
using Distributed
addprocs(4)

@everywhere include("run_h.jl")
@everywhere using .run_h

run_h_v3(1:5)

```

I get again the error:

```julia
On worker 2:
UndefVarError: A not defined

```

This must be related to the scope of the variables, but I don’t exactly what is going on.

It gets even weirder if I make the size of the matrix a parameter of the function `run_h`. In IJulia:

```julia
using Distributed
addprocs(2)

@everywhere using LinearAlgebra, Random
@everywhere function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v4(n, iter)
    @everywhere A = zeros(n, n)
    return pmap(x->h!(A, x), iter)
end

run_h_v4(100, 1:50)

```

I get the error:

```julia
On worker 2:
UndefVarError: n not defined

```

So the argument `n` is not available to the `@everywhere ...`. I don’t understant why this should happen.

---

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 12, 2019, 5:44pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/6 "2019-06-12T17:44:41Z")

</div>

An update: using `sendto` from `ParallelDataTransfer.jl`, instead of `@everywhere ...` appears to solve the problems. Defining the new version of the function:

```julia
function run_h_v5(n, iter)
    A = zeros(n, n)
    sendto(workers(), A = A)
    return pmap(x->h!(A, x), iter)
end

```

and then running `run_h_v5(100, 1:50)` works with no problems, even if placed inside a module. However, I don’t understand what `sendto` is doing that `@everywhere` does not. Any ideas?

---

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 12, 2019, 6:10pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/7 "2019-06-12T18:10:51Z")

</div>

Digging a little further, it appears that this issue when passing the argument `n` to  
`@everywhere A = zeros(n, n)`  
has been discussed before at [https://github.com/JuliaLang/julia/issues/9118](https://github.com/JuliaLang/julia/issues/9118) and is document at [Distributed Computing · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.@everywhere)

The solution is to use string interpolation with `@everywhere`:

```julia
function run_h_v6(n, iter)
    @everywhere A = $(zeros(n, n))
    return pmap(x->h!(A, x), iter)
end

```

However, the code in the original post still throws the error `UndefVarError: A not defined` when I run `test2()`, even if I use string interpolation to initialize `A`.

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [June 12, 2019, 11:44pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/8 "2019-06-12T23:44:56Z")

</div>

Note that `@everywhere` executes the statement under module `Main` on the child process; this is causing the error when you move the code to module `run_h`. More on this [here](https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.@everywhere) and [here](https://docs.julialang.org/en/v1/manual/parallel-computing/#Code-Availability-and-Loading-Packages-1).

In general, it is better to keep the functions self sufficient, and not have them rely on a global variable that is initialised from elsewhere. If you really insist on having a module-scoped global matrix `A` (which I am not at all sure you need, btw), it can be initialised lazily. Smth. like this:

```julia
# my_mod.jl
module MyMod

using Distributed, LinearAlgebra, Random

A = zeros(0,0)

function h!(x)
    global A
    if length(A) == 0
        A = zeros(100, 100)
    end

    return eigvals(rand!(MersenneTwister(x), A))

end

function run_h(iter)
    if nprocs() > 1
        @everywhere include("my_mod.jl") # See more details below
    end
    return pmap(MyMod.h!, iter) 
end

export run_h

end

```

And then in REPL:

```julia
julia> include("my_mod.jl")

julia> using .MyMod, Distributed

julia> run_h(1:20) # works

julia> addprocs(4)

julia> run_h(1:20) # also works

```

See also [this SO post](https://stackoverflow.com/questions/47685536/how-do-you-load-a-module-everywhere-inside-a-function-in-julia) on how to load the code on the workers with `@everywhere using ...`, if your module code is distributed via the package manager (apparently you need to mess with macros to achieve that). As per [addprocs doc](https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.addprocs), the workers inherit the working directory and julia binary of the master process, unless explicitly specified otherwise. JULIA\_PROJECT is specified via the shell environment.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 13, 2019, 10:41am UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/9 "2019-06-13T10:41:05Z")

</div>

Thank you once again for your comments [tanhevg](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/8)!

> If you really insist on having a module-scoped global matrix `A` (which I am not at all sure you need, btw)

This is exactly what I want to avoid doing. I do not want to have a matrix `A` defined to the whole module. I want the matrix `A` to “exist” only inside the function `run_h`, but while I want `A` to be defined only inside the function `run_h`, I want it to be available to all processes (if this makes sense).

I have found a solution that uses `@spawnat` instead of `@everywhere` that works. Going back to the original problem, if I write:

```julia
using Distributed
addprocs(4)
@everywhere using LinearAlgebra

function test3()
    if nprocs()>1
        for p in procs()
            @spawnat p A = zeros(100, 100)
        end
        return eigvals(A)
    else
        A = zeros(100, 100)
        return eigvals(A)
    end
end

```

`test3()` works whether I add procs or not.

For the less minimal example `run_h`, this version uses map/pmap dependending on the number of available procs (althought I could just always use pmap) and works for any number of procs:

```julia
using Distributed
addprocs(4)
@everywhere using LinearAlgebra, Random

@everywhere function h!(mat, x) 
    return eigvals(rand!(MersenneTwister(x), mat))
end

function run_h(n, iter)
    if nprocs()>1
        for p in procs()
            @spawnat p A = zeros(n, n)
        end
        return pmap(x->h!(A, x), iter)
    else
        A = zeros(n, n)
        return map(x->h!(A, x), iter)
    end
end

```

If I understand correctly `ParallelDataTransfer.jl` uses `@spawnat` to distribute variables and so that is why `run_h_v5(n, iter)` also worked.

For my particular problem, I am happy enough with what I have now. However, I would still like to understand why `test2()` would fail while `test1()` works, when `nprocs()>1`.

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [June 13, 2019, 11:52am UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/11 "2019-06-13T11:52:32Z")

</div>

There are a few problems with the examples.

1. `@spawnat` creates a potential race condition, because it runs asynchronously, so the remote statement is not guaranteed to complete before the next line of code (or iteration of the for loop) executes. To avoid the race, do something like `f = @spawnat p ...; wait(f)`.

2. You don’t need the `@spawnat` anyway. Consider this:

```julia
function run_h(n, iter)
    A = zeros(n, n)
    return pmap(x->h!(A, x), iter)
end

```

This works regardless of the number of workers (i.e. whether `addprocs()` is called in the beginning or not). The matrices that are created by `@spawnat` are not used within `pmap` - this can be verified by modifying the contents of local `A` before `pmap` is called and printing `A` from `h!`

I think what is really happening here is this: when the anonymous inner function is created (`x->h!(A,x)`), it captures all the necessary local variables (in this case `A`). This inner function, along with all the captured variables, is shipped to the workers and executed there. So, if `A` is very big, this can be quite expensive. But this achieves your intentions: `A` exists only inside `run_h`.

---

<div class="post-metadata">

**Author:** ![Bruno\_Amorim](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@Bruno\_Amorim](https://discourse.julialang.org/u/Bruno_Amorim)\
**Post date:** [June 13, 2019, 2:01pm UTC](https://discourse.julialang.org/t/undefvarerror-when-initializing-variable-inside-function-with-everywhere/25183/12 "2019-06-13T14:01:31Z")

</div>

Thank you very much! You are correct, indeed the matrices created by @spawant are never used by pmap. Furthermore, and this is something I was missing before, both @everywhere, @spawnat and sendto from ParallelDataTransfer.jl, when called from inside a function, define A not in the function local scope, but in Main.

This example shows this. I defined a module:

```julia
module run_h
using Distributed, Random, LinearAlgebra, ParallelDataTransfer

export run_h_check

function h!(mat, x) 
    
    println("Matrix inside pmap ", myid(), " A=", mat)
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_check(n, iter)
    A = zeros(n, n) # initialized A
    println("original A")
    println(A)
    sendto(workers(), A = A) # make A available to other processes, this actually exports A to Main
    @everywhere workers() A[1, 1] = myid()
    println("Check that A was modified in the workers")
    @everywhere workers() println(A)
    println("Pmap...")
    return pmap(x->h!(A, x), iter)
end

end

```

And then I run fro IJulia:

```julia
using Distributed
addprocs(4)

@everywhere include("run_h.jl")
@everywhere using .run_h

run_h_check(2, 1:5)

```

which returns:

```julia
original A
[0.0 0.0; 0.0 0.0]
Check that A was modified in the workers
      From worker 3:	[3.0 0.0; 0.0 0.0]
      From worker 5:	[5.0 0.0; 0.0 0.0]
      From worker 2:	[2.0 0.0; 0.0 0.0]
      From worker 4:	[4.0 0.0; 0.0 0.0]
Pmap...
      From worker 4:	Matrix inside pmap 4 A=[0.0 0.0; 0.0 0.0]
      From worker 5:	Matrix inside pmap 5 A=[0.0 0.0; 0.0 0.0]
      From worker 3:	Matrix inside pmap 3 A=[0.0 0.0; 0.0 0.0]
      From worker 2:	Matrix inside pmap 2 A=[0.0 0.0; 0.0 0.0]
      From worker 4:	Matrix inside pmap 4 A=[0.0 0.0; 0.0 0.0]
5-element Array{Array{Float64,1},1}:
 [0.470351, -0.226408] 
 [0.191387, 0.994748]  
 [-0.00607352, 1.78786]
 [-0.102802, 1.71222]  
 [0.191155, 1.22056]

```

Indeed, inside the pmap, only the original A=[0.0 0.0; 0.0 0.0] is used. I can then check that `sendto` actually moved the matrix A to Main. If I run:  
`@everywhere workers() println(Main.A)`  
I get:

```julia
      From worker 4:	A= [4.0 0.0; 0.0 0.0]
      From worker 2:	A= [2.0 0.0; 0.0 0.0]
      From worker 3:	A= [3.0 0.0; 0.0 0.0]
      From worker 5:	A= [5.0 0.0; 0.0 0.0]

```

But if I simply run:  
`println(Main.A)`, I obtain the error `UndefVarError: A not defined`, since `A` was not exported to Main of process 1.

I wasn’t expecting for the matrix A to be automatially made available to all processes in pmap. This is probably the same behaviour that is discussed in [@everywhere and pmap inside of a function?](https://discourse.julialang.org/t/everywhere-and-pmap-inside-of-a-function/5405)

Once again, thank you very much [tanhevg](https://discourse.julialang.org/u/tanhevg)! This was really useful!