Precompilation, environments, and `Distributed` computing

General Usage
, , ,
1

I’m using Distributed.jl to run simulations in parallel from a Jupyter Notebook, and my simulations are performed using a local repo SimModels (I have not yet made it a module) which pmaps various models/simulations to each processor from Distributed. I’m encountering similar issues to this question and am hoping for some guidance on how to setup my workflow.

Right now, I’m:

  1. Setting Distributed environment by: using Distributed; addprocs(20)
  2. Setting up the environment:
    @everywhere using Pkg; @everywhere Pkg.activate("SimEnv"); @everywhere Pkg.instantiate()
  3. Including my code everywhere: @everywhere include("..path-to-code/SimModels.jl")

But when doing this, I am getting errors from the third step like

From worker 14:	│   exception = Required dependency Compat [34da2185-b29b-5c13-b0c7-acf172513d20] failed to load from a cache file.

Ideally I would be able to run a setup.jl script on a single processor which sets the whole Distributed environment, precompiles the necessary packages once, and then loads my local code on each processor in a way that uses the precompiled code.

Am I thinking about this correctly, or how might I go about this in a better way? Thanks!

1 Like
2

Did you find a working solution for this? I have similar problems, in such a way as that each worker tends to precompile all packages of the environment. This fails, presumably because they are all trying to do it at the same time.

Sometimes I get it to work, though I haven’t found out yet exactly how.

1 Like