Pkg.instantiate() takes a very long time on a remote worker

Specific Domains Julia at Scale
1

I’m trying to use multiple nodes for a parallel algorithm, but currently Pkg.instantiate() takes far too long, as in 20+ minutes, so long that my 30 minute job timed out. Is there a way to speed this up? Is this normal?

2

So I actually have 2 problems. The first, related to the above, is that my workers do not inherit the correct DEPOT_PATH variable (I have to change this so as to not install julia packages in my user folder on the HPC I use, since we have a “data” folder we should use instead. I have now added:

Distributed.@everywhere begin 
    DEPOT_PATH[1] = "/data/leuven/331/vsc33168/.julia"
    @info "Set DEPOT_PATH = $DEPOT_PATH"
    using Pkg
    Pkg.activate(dirname(@__FILE__))
    Pkg.instantiate()

    include(<path_to_script_containing_functions>)

The second is that Julia does not wait for Pkg.instantiate() to finish before continuing my script and calling using <pkg>, which leads to errors.

1 Like
3

Both of these errors were fixed by setting the DEPOT_PATH variable on the workers. Thank you me :stuck_out_tongue: