Distributed.jl to run simulations in parallel from a Jupyter Notebook, and my simulations are performed using a local repo
SimModels (I have not yet made it a module) which
pmaps various models/simulations to each processor from
Distributed. I’m encountering similar issues to this question and am hoping for some guidance on how to setup my workflow.
Right now, I’m:
- Setting Distributed environment by:
using Distributed; addprocs(20)
- Setting up the environment:
@everywhere using Pkg; @everywhere Pkg.activate("SimEnv"); @everywhere Pkg.instantiate()
- Including my code everywhere:
But when doing this, I am getting errors from the third step like
From worker 14: │ exception = Required dependency Compat [34da2185-b29b-5c13-b0c7-acf172513d20] failed to load from a cache file.
Ideally I would be able to run a
setup.jl script on a single processor which sets the whole
Distributed environment, precompiles the necessary packages once, and then loads my local code on each processor in a way that uses the precompiled code.
Am I thinking about this correctly, or how might I go about this in a better way? Thanks!