Error running distributed code inside of a module

General Usage
7

Amazing, thank you!

For posterity (and those Googling who end up landing here in the future), here’s a short write-up.

After a bit of experimentation, it seems like @everywhere procs ..., for the initalisation step, is the secret sauce here. It works when for proc in procs; @spawnat proc ...; end does not. I’ve not tried digging into why that is.

By-the-by one needs to include a couple of extra lines to activate the current environment inside the new worker processes. I’ve tested that the following works when installed as a package called MyPackage:

# MyPackage.jl

module MyPackage

import Distributed
import Pkg

function test(procs)
    project_path = splitdir(Pkg.project().path)[1]
    Distributed.@everywhere procs begin
        Main.eval(quote
            import Pkg
            Pkg.activate($$project_path)
			import MyPackage
        end)
    end

    Distributed.pmap(_->println("Hello from " * string(Distributed.myid())), Distributed.WorkerPool(procs), 1:6)
end

function test(num_procs::Integer)
    project_path = splitdir(Pkg.project().path)[1]
    procs = Distributed.addprocs(num_procs)
    try
        test(procs)
    finally
        Distributed.rmprocs(procs)
    end
end

end

In each case the complicated bit being the initial set-up on each worker. After that you’re free to use pmap, @spawnat or @everywhere as you prefer.

I’ve not tested every combination of [my code above, @MilesCranmer's code above] x [environment, no environment] x [package, no package] and the behaviour in each case. I’m pretty sure sometimes it’s different. YMMV.

5 Likes