Making code and packages available to workers inside module

Nikos_Gianniotis · March 2, 2021, 8:35am

Dear forum, my title explains to some extend what I am trying to achieve.

To make it hopefully clearer, I have written the following minimal code as an example.

First, I create an ordinary script (scriptA.jl) where the desired functionality is implemented:
(1) there is a function called forallworkers that is made available to all available workers
(2) there is a second function called caller, available only to the process handling the REPL (is this the right way of putting it?) which calls function forallworkers.

In my real problem, function forallworkers implements some tedius and lengthy calculations and function caller calls forallworkers multiple times in parallel using pmap while doing some extra calculations which are not relevant here. Hence, the user can comfortably call function caller while being oblivious to the fact that there are computations that run in parallel.

If I start julia 1.5.3 with julia -O3 -p2, load the script with include("scriptA.jl") below

# scriptA.jl
# starting julia 1.5.3 with: julia -O3 -p2

@everywhere using SpecialFunctions, Printf

@everywhere function forallworkers(x)
    @printf("Result from worker %d is %.3f\n", myid(), erf(x))
    return erf(x)
end

function caller(X)
    # some computation on X takes place here in the real problem, 
    # but is not relevant to this MWE
    pmap(forallworkers, X)
end

and call caller(1:5), things will run as expected:

julia> caller(1:5)
      From worker 3:	Result from worker 3 is 0.995
      From worker 2:	Result from worker 2 is 0.843
      From worker 3:	Result from worker 3 is 1.000
      From worker 2:	Result from worker 2 is 1.000
      From worker 3:	Result from worker 3 is 1.000
5-element Array{Float64,1}:
 0.8427007929497149
 0.9953222650189527
 0.9999779095030014
 0.9999999845827421
 0.9999999999984626

Now, I would like to put this functionality in a module (with the future intention of creating a package). My first attempt was:

# scriptB.jl
# starting julia 1.5.3 with: julia -O3 -p2

module modMWE

    export caller

    @everywhere using SpecialFunctions, Printf

    @everywhere function forallworkers(x)
        @printf("Result from worker %d is %.3f\n", myid(), erf(x))
        return erf(x)

    end

    function caller(X)
        # some computation on X takes place here in the real problem, 
        # but is not relevant to this MWE
        pmap(forallworkers, X)
    end

end

However, if I load scriptB.jl immediately after starting julia with include("scriptB.jl"), I get the message

"ERROR: LoadError: LoadError: UndefVarError: @everywhere not defined"

which I found odd since I started julia with two processes using the flag -p2. Nevertheless, after inserting the line using Distributed the problem seems to go away in scriptC.jl:

# scriptC.jl
# starting julia 1.5.3 with: julia -O3 -p2

module modMWE

    export caller

    using Distributed # <------ newly added line

    @everywhere using SpecialFunctions, Printf

    @everywhere function forallworkers(x)
        @printf("Result from worker %d is %.3f\n", myid(), erf(x))
        return erf(x)
    end

    function caller(X)
        # some computation on X takes place here in the real problem, 
        # but is not relevant to this MWE
        pmap(forallworkers, X)
    end

end

Now I can successfully load the module with include("scriptC.jl"). However, if I now try and call function caller with modMWE.caller(1:5) this results in the following error:

julia> modMWE.caller(1:5)
ERROR: UndefVarError: forallworkers not defined
Stacktrace:
 [1] caller(::UnitRange{Int64}) at ./REPL[1]:15
 [2] top-level scope at REPL[2]:1

My naive impression is that function forallworkers should be available to all processes since its definition has been qualified with @everywhere, but obviously this is not the case.

My question therefore:

How can I properly create a module that has functions that visible to all workers so that they can be run in parallel?
I.e. in this particular example this specifically translates to, how do I make function forallworkers properly available to all workers and callable within function caller?

Many thanks!

EDIT: After solving the problem, I add here how the final version of the script (you can also find it below if you follow the discussion) that solves the problem.

# scriptE.jl
# starting julia 1.5.3 with: julia -O3 -p2

@everywhere module modMWE

    export caller

    using Distributed

    using SpecialFunctions, Printf

    function forallworkers(x)
        @printf("Result from worker %d is %.3f\n", myid(), erf(x))
        return erf(x)
    end

    function caller(X)
        # some computation on X takes place here in the real problem,
        # but is not relevant to this MWE
        pmap(forallworkers, X)
    end

end

marius311 · March 2, 2021, 8:52am

The key part of the docs for @everywhere to catch is “Execute an expression under Main”. So in scriptC when you have those @everywhere’s inside your module, that code isn’t evaluated in your module, its evaluated in Main, so you end up defining Main.forallworkers on all processes. You can see that even on the main process, modMWE.forallworkers isn’t defined.

The two things you can do are 1) remove the @everywheres from inside the module and evaluate the entire module definition on all workers, so @everywhere module modMWE .... Then modMWE.caller(1:5) will work, or (better) 2) turn this into a bonafide package with a Project.toml and src/ directory, etc…, then using modMWE automatically loads the module on all workers, without the need for @everywheres anywhere.

Edit: I should also say, welcome, and thanks for posting a really clear MWE!

Nikos_Gianniotis · March 2, 2021, 10:37am

@marius311 Thanks for the reply and the warm welcome!

I had forgotten that @everywhere should be executed under Main, so this is helpful already. Now I understand why modMWE.forallworkers is not visible to function caller, as you explain.

However, I am still not entirely clear how I should proceed despite your suggestions. In my attempt to follow your suggestion, I have now created a new script called scriptD.jl which reads as follows:

# scriptD.jl
# starting julia 1.5.3 with: julia -O3 -p2

@everywhere module modMWE

    export caller

    using Distributed

    using SpecialFunctions, Printf

    function forallworkers(x)
        @printf("Result from worker %d is %.3f\n", myid(), erf(x))
        return erf(x)
    end

    function caller(X)
        # some computation on X takes place here in the real problem, 
        # but is not relevant to this MWE
        map(forallworkers, X)
    end

end

I load the script again with include("scriptD.jl"). If I now call my function caller, I get the following output on the REPL:

julia> modMWE.caller(1:5)
Result from worker 1 is 0.843
Result from worker 1 is 0.995
Result from worker 1 is 1.000
Result from worker 1 is 1.000
Result from worker 1 is 1.000
5-element Array{Float64,1}:
 0.8427007929497149
 0.9953222650189527
 0.9999779095030014
 0.9999999845827421
 0.9999999999984626

Repeating the call, I seem to always get the exact same output: worker 1 always shows up, but other workers never do, despite Julia informing that there are indeed two workers available:

julia> nworkers()
2

What I understand, is that even though function caller must be available to all workers, me calling it from the REPL results in only worker 1 doing the work because it is the worker that is “responsible” for the REPL.

If I explicitly call things in parallel, like:

pmap(modMWE.caller, 1:5)
      From worker 2:	Result from worker 2 is 0.843
      From worker 3:	Result from worker 3 is 0.995
      From worker 3:	Result from worker 3 is 1.000
      From worker 2:	Result from worker 2 is 1.000
      From worker 3:	Result from worker 3 is 1.000
5-element Array{Float64,1}:
 0.8427007929497149
 0.9953222650189527
 0.9999779095030014
 0.9999999845827421
 0.9999999999984626

this works fine.

But: how can I make function caller internally distribute the work to other workers without me having to call things explicitly in parallel like above with e.g. pmap(caller, 1:5).

Many thanks in advance!

marius311 · March 2, 2021, 11:37am

Looks like you switched caller from using pmap to using map, just keep it as pmap.

Nikos_Gianniotis · March 2, 2021, 11:49am

@marius311 Sorry for this typo!

Below the final script that works indeed as expected:

# scriptE.jl
# starting julia 1.5.3 with: julia -O3 -p2

@everywhere module modMWE

    export caller

    using Distributed

    using SpecialFunctions, Printf

    function forallworkers(x)
        @printf("Result from worker %d is %.3f\n", myid(), erf(x))
        return erf(x)
    end

    function caller(X)
        # some computation on X takes place here in the real problem,
        # but is not relevant to this MWE
        pmap(forallworkers, X)
    end

end

Load with include("scriptE.jl") and call:

julia> modMWE.caller(1:5)
      From worker 2:	Result from worker 2 is 0.843
      From worker 3:	Result from worker 3 is 0.995
      From worker 3:	Result from worker 3 is 1.000
      From worker 2:	Result from worker 2 is 1.000
      From worker 3:	Result from worker 3 is 1.000
5-element Array{Float64,1}:
 0.8427007929497149
 0.9953222650189527
 0.9999779095030014
 0.9999999845827421
 0.9999999999984626

Let me see if I understand why the above works: qualifying the module definition with @everywhere makes the module available to all workers. As the module is available to all workers, so are its functions. It is as simple as that, right?