UndefVarError when initializing variable inside function with @everywhere

Specific Domains Julia at Scale
1

I am trying to solve a problem in which I have to evaluate a function, f(x), for several different values of an argument x. I want to write a function that depending on the number of available processes it uses either map (if nprocs()==1) or pmap (if nprocs()>1). The function f(x) I want to evaluate requires a preallocated matrix and this is where I find a behaviour I do not understand.

This minimal example shows the problem I am facing:

using Distributed
addprocs(4)
@everywhere using LinearAlgebra

function test1()
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return eigvals(A)
    else
        B = zeros(100, 100)
        return eigvals(B)
    end
end

function test2()
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return eigvals(A)
    else
        A = zeros(100, 100)
        return eigvals(A)
    end
end

If I run test1(), it works without problems. However, when I run test2(), I get the error: UndefVarError: A not defined

If I run the code without adding any procs, both test1() and test2() work. Of course I can just write my code in the style of test1(), but I would like to understand why test2() fails. Thanks in advance for any suggestion!

EDIT: Changed the title of the topic to reflect the fact that the issue seems to be unrelated to the use of and if clause

2

In general this seems like a bug: the if parts of if/else clause of the two methods seem completely identical to me (unless I am missing a trick), so it is not clear why the else part that does not get executed would cause an error.

To recap what is going on here, however, with addprocs this code spawns four child processes, and the @everywhere ... statement creates a global variable A both on parent and on all children. The calls to eigval, that are not prefixed by @spawn or any other macro from Distributed, will execute on master only. Is this really the intention?

3

To give some more context:
In the actual problem I am trying to solve, I will run the eigvals function inside a map/pmap. So a slightly less minimal example would be:

using Distributed
addprocs(4)

@everywhere using Random, LinearAlgebra

# function which I want to run using map/pmap (it fills a preallocated matrix, mat, with random numbers and computes its eigenvalues) for different values of x.
@everywhere function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end


function run_h_v1(iter)
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return pmap(x->h!(A, x), iter)
    else
        B = zeros(100, 100)
        return map(x->h!(B, x), iter)
    end
end


function run_h_v2(iter)
    if nprocs()>1
        @everywhere A = zeros(100, 100)
        return pmap(x->h!(A, x), iter)
    else
        A = zeros(100, 100)
        return map(x->h!(A, x), iter)
    end
end

So in order to avoid allocating a new matrix for each value of x, I preallocate a matrix A, which is modified by the function h!(mat, x). When several processes are available, I initialize the A matrix in all processes using @everywhere.

Now, run_h_v1(1:50) runs fine, while run_h_v2(1:50) throws the same error as the example in the original post:

On worker 2:
UndefVarError: A not defined

As the example in the original post indicates, the error seems to caused by the use of @everywhere inside an if. Is this really a bug or I am missing something?

P.S.: I can probably just always use pmap for both cases (nproc()==1 and nproc()>1) as it appears there is no overhead in using pmap instead of map. But I would still like to understand what is the origin of this issue.

4

Just pre-allocate A outside of the function. The @everywhere macro executes just the statement that is passed to it, so A will be in global scope on the child processes anyway.

The code below works, regardless of whether addprocs were called or not.

using Distributed
addprocs(4)
@everywhere using LinearAlgebra, Random

@everywhere A = zeros(100, 100)

@everywhere function h!(x)

    return eigvals(rand!(MersenneTwister(x), A))

end

function run_h(iter)
    if nprocs()>1
        return pmap(x->h!(x), iter)
    else
        return map(x->h!(x), iter)
    end
end
5

Thank you tanhevg for your suggestion! Indeed that works, but I would like for the matrix A to be defined only when the function run_h is called (I want to include the code inside a module and the size of the matrix A might always not be the same).

But there is something about the use of @everywhere inside a function that I am probably missing and the error I am getting might be unrelated to the use of an if clause. For example the following code works fine in an interactive environment (IJulia):

using Distributed

addprocs(2)

@everywhere using LinearAlgebra, Random

@everywhere function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v3(iter)
    @everywhere A = zeros(100, 100)
    return pmap(x->h!(A, x), iter)
end

run_h_v3(1:20)

However, if I move the code to a module run_h.jl:

module run_h

using Distributed, Random, LinearAlgebra

export run_h_v3

function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v3(iter)
    @everywhere A = zeros(100, 100)
    return pmap(x->h!(A, x), iter)
end

end

and then do in an interactive environment

using Distributed
addprocs(4)

@everywhere include("run_h.jl")
@everywhere using .run_h

run_h_v3(1:5)

I get again the error:

On worker 2:
UndefVarError: A not defined

This must be related to the scope of the variables, but I don’t exactly what is going on.

It gets even weirder if I make the size of the matrix a parameter of the function run_h. In IJulia:

using Distributed
addprocs(2)

@everywhere using LinearAlgebra, Random
@everywhere function h!(mat, x) 
    
    return eigvals(rand!(MersenneTwister(x), mat))
    
end

function run_h_v4(n, iter)
    @everywhere A = zeros(n, n)
    return pmap(x->h!(A, x), iter)
end

run_h_v4(100, 1:50)

I get the error:

On worker 2:
UndefVarError: n not defined

So the argument n is not available to the @everywhere .... I don’t understant why this should happen.

6

An update: using sendto from ParallelDataTransfer.jl, instead of @everywhere ... appears to solve the problems. Defining the new version of the function:

function run_h_v5(n, iter)
    A = zeros(n, n)
    sendto(workers(), A = A)
    return pmap(x->h!(A, x), iter)
end

and then running run_h_v5(100, 1:50) works with no problems, even if placed inside a module. However, I don’t understand what sendto is doing that @everywhere does not. Any ideas?

7

Digging a little further, it appears that this issue when passing the argument n to
@everywhere A = zeros(n, n)
has been discussed before at https://github.com/JuliaLang/julia/issues/9118 and is document at Distributed Computing · The Julia Language

The solution is to use string interpolation with @everywhere:

function run_h_v6(n, iter)
    @everywhere A = $(zeros(n, n))
    return pmap(x->h!(A, x), iter)
end

However, the code in the original post still throws the error UndefVarError: A not defined when I run test2(), even if I use string interpolation to initialize A.

8

Note that @everywhere executes the statement under module Main on the child process; this is causing the error when you move the code to module run_h. More on this here and here.

In general, it is better to keep the functions self sufficient, and not have them rely on a global variable that is initialised from elsewhere. If you really insist on having a module-scoped global matrix A (which I am not at all sure you need, btw), it can be initialised lazily. Smth. like this:

# my_mod.jl
module MyMod

using Distributed, LinearAlgebra, Random

A = zeros(0,0)

function h!(x)
    global A
    if length(A) == 0
        A = zeros(100, 100)
    end

    return eigvals(rand!(MersenneTwister(x), A))

end

function run_h(iter)
    if nprocs() > 1
        @everywhere include("my_mod.jl")  # See more details below
    end
    return pmap(MyMod.h!, iter) 
end

export run_h


end

And then in REPL:

julia> include("my_mod.jl")

julia> using .MyMod, Distributed

julia> run_h(1:20) # works

julia> addprocs(4)

julia> run_h(1:20) # also works

See also this SO post on how to load the code on the workers with @everywhere using ..., if your module code is distributed via the package manager (apparently you need to mess with macros to achieve that). As per addprocs doc, the workers inherit the working directory and julia binary of the master process, unless explicitly specified otherwise. JULIA_PROJECT is specified via the shell environment.

Hope this helps.

9

Thank you once again for your comments tanhevg!

If you really insist on having a module-scoped global matrix A (which I am not at all sure you need, btw)

This is exactly what I want to avoid doing. I do not want to have a matrix A defined to the whole module. I want the matrix A to “exist” only inside the function run_h, but while I want A to be defined only inside the function run_h, I want it to be available to all processes (if this makes sense).

I have found a solution that uses @spawnat instead of @everywhere that works. Going back to the original problem, if I write:

using Distributed
addprocs(4)
@everywhere using LinearAlgebra

function test3()
    if nprocs()>1
        for p in procs()
            @spawnat p A = zeros(100, 100)
        end
        return eigvals(A)
    else
        A = zeros(100, 100)
        return eigvals(A)
    end
end

test3() works whether I add procs or not.

For the less minimal example run_h, this version uses map/pmap dependending on the number of available procs (althought I could just always use pmap) and works for any number of procs:

using Distributed
addprocs(4)
@everywhere using LinearAlgebra, Random

@everywhere function h!(mat, x) 
    return eigvals(rand!(MersenneTwister(x), mat))
end

function run_h(n, iter)
    if nprocs()>1
        for p in procs()
            @spawnat p A = zeros(n, n)
        end
        return pmap(x->h!(A, x), iter)
    else
        A = zeros(n, n)
        return map(x->h!(A, x), iter)
    end
end

If I understand correctly ParallelDataTransfer.jl uses @spawnat to distribute variables and so that is why run_h_v5(n, iter) also worked.

For my particular problem, I am happy enough with what I have now. However, I would still like to understand why test2() would fail while test1() works, when nprocs()>1.

11

There are a few problems with the examples.

  1. @spawnat creates a potential race condition, because it runs asynchronously, so the remote statement is not guaranteed to complete before the next line of code (or iteration of the for loop) executes. To avoid the race, do something like f = @spawnat p ...; wait(f).

  2. You don’t need the @spawnat anyway. Consider this:

function run_h(n, iter)
    A = zeros(n, n)
    return pmap(x->h!(A, x), iter)
end

This works regardless of the number of workers (i.e. whether addprocs() is called in the beginning or not). The matrices that are created by @spawnat are not used within pmap - this can be verified by modifying the contents of local A before pmap is called and printing A from h!

I think what is really happening here is this: when the anonymous inner function is created (x->h!(A,x)), it captures all the necessary local variables (in this case A). This inner function, along with all the captured variables, is shipped to the workers and executed there. So, if A is very big, this can be quite expensive. But this achieves your intentions: A exists only inside run_h.

12

Thank you very much! You are correct, indeed the matrices created by @spawant are never used by pmap. Furthermore, and this is something I was missing before, both @everywhere, @spawnat and sendto from ParallelDataTransfer.jl, when called from inside a function, define A not in the function local scope, but in Main.

This example shows this. I defined a module:

module run_h
using Distributed, Random, LinearAlgebra, ParallelDataTransfer

export run_h_check

function h!(mat, x) 
    
    println("Matrix inside pmap ", myid(), " A=", mat)
    return eigvals(rand!(MersenneTwister(x), mat))
    
end


function run_h_check(n, iter)
    A = zeros(n, n) # initialized A
    println("original A")
    println(A)
    sendto(workers(), A = A) # make A available to other processes, this actually exports A to Main
    @everywhere workers() A[1, 1] = myid()
    println("Check that A was modified in the workers")
    @everywhere workers() println(A)
    println("Pmap...")
    return pmap(x->h!(A, x), iter)
end

end

And then I run fro IJulia:

using Distributed
addprocs(4)

@everywhere include("run_h.jl")
@everywhere using .run_h

run_h_check(2, 1:5)

which returns:

original A
[0.0 0.0; 0.0 0.0]
Check that A was modified in the workers
      From worker 3:	[3.0 0.0; 0.0 0.0]
      From worker 5:	[5.0 0.0; 0.0 0.0]
      From worker 2:	[2.0 0.0; 0.0 0.0]
      From worker 4:	[4.0 0.0; 0.0 0.0]
Pmap...
      From worker 4:	Matrix inside pmap 4 A=[0.0 0.0; 0.0 0.0]
      From worker 5:	Matrix inside pmap 5 A=[0.0 0.0; 0.0 0.0]
      From worker 3:	Matrix inside pmap 3 A=[0.0 0.0; 0.0 0.0]
      From worker 2:	Matrix inside pmap 2 A=[0.0 0.0; 0.0 0.0]
      From worker 4:	Matrix inside pmap 4 A=[0.0 0.0; 0.0 0.0]
5-element Array{Array{Float64,1},1}:
 [0.470351, -0.226408] 
 [0.191387, 0.994748]  
 [-0.00607352, 1.78786]
 [-0.102802, 1.71222]  
 [0.191155, 1.22056]

Indeed, inside the pmap, only the original A=[0.0 0.0; 0.0 0.0] is used. I can then check that sendto actually moved the matrix A to Main. If I run:
@everywhere workers() println(Main.A)
I get:

      From worker 4:	A= [4.0 0.0; 0.0 0.0]
      From worker 2:	A= [2.0 0.0; 0.0 0.0]
      From worker 3:	A= [3.0 0.0; 0.0 0.0]
      From worker 5:	A= [5.0 0.0; 0.0 0.0]

But if I simply run:
println(Main.A), I obtain the error UndefVarError: A not defined, since A was not exported to Main of process 1.

I wasn’t expecting for the matrix A to be automatially made available to all processes in pmap. This is probably the same behaviour that is discussed in @everywhere and pmap inside of a function?

Once again, thank you very much tanhevg! This was really useful!