Writing a module using paralellization with @everywhere and @parallel

General Usage
,
1

Hi,
I am rather new to julia and am trying to speed up some processes by using @parallel . But somehow I get confused with the correct way of including a (possible) parallelization in a module.

If I use @everywhere inside of a module environment, then it’s not found when including the module with using, see MWE2 below. (For reference, the code runs fine without the module environment and by simply including the module file.)

Next I understood I simply do not use @everywhere inside the module and instead using @everywhere include("...") and @everywhere using ... in the main script. While that seemingly works, I get a bunch of Warnings when including other modules, see MWE1 below`.

Hence, I probably did not understand the correct way of including parallelization in a module. Could somebody give me a short explanation on how to do that maybe using the examples here?

Best,
Tim

PS: Yes, I do know there are more effective ways to double the value of an array, the examples below are just meant as illustration (:

MWE1
MyMain_parallel.jl

@everywhere include("MyModFile_parallel.jl")
println("included MyModFile_parallel.jl")

@everywhere using MyMod: double_it, MyType
println("loaded MyMod")

x_in = MyType[1 3; 5 7]
x_out = SharedArray{MyType}(size(x_in))

println(x_in)
println(x_out)
double_it(x_in, x_out)
println(x_out)

MyMod_parallel.jl

println("inside MyModFile_parallel.jl")

module MyMod
using NearestNeighbors # just loading for testing reasons

MyType = Float64

function double_it_worker(x::MyType)
    2*x
end

function double_it(in_array::Array{MyType}, out_array::SharedArray{MyType})
    # do some random stuff with @parallel
    @assert size(in_array) == size(out_array)
    @sync @parallel for i = 1:length(in_array)
        out_array[i] = double_it_worker(in_array[i])
    end
end

end

MWE2
MyMain_parallel2.jl

include("MyModFile_parallel2.jl")
println("included MyModFile_parallel2.jl")

using MyMod: double_it, MyType
println("loaded MyMod")

x_in = MyType[1 3; 5 7]
x_out = SharedArray{MyType}(size(x_in))

println(x_in)
println(x_out)
double_it(x_in, x_out)
println(x_out)

MyMod_parallel2.jl

println("inside MyModFile_parallel2.jl")

module MyMod
using NearestNeighbors # just loading for testing reasons

@everywhere MyType = Float64

@everywhere function double_it_worker(x::MyType)
    2*x
end

@everywhere function double_it(in_array::Array{MyType}, out_array::SharedArray{MyType})
    # do some random stuff with @parallel
    @assert size(in_array) == size(out_array)
    @sync @parallel for i = 1:length(in_array)
        out_array[i] = double_it_worker(in_array[i])
    end
end

end
2

I have the same question.