Activating virtual environment on workers

martincornejo · August 12, 2020, 1:02pm

Say we want to use certain package (that is only available in a project-specific virtual environment) in a distributed manner.

(@v1.5) pkg> activate .
 Activating environment at `C:\Users\Martin Cornejo\MyProject\Project.toml`

julia> using Distributed

julia> addprocs(1)
1-element Array{Int64,1}:
 2

julia> @everywhere using JuMP    # for example
ERROR: On worker 2:
ArgumentError: Package JuMP [4076af6c-e467-56ae-b986-b466b2749572] is 
required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.      

_require at .\loading.jl:999
require at .\loading.jl:928
#1 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\Distributed.jl:78
#103 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:290
run_work_thunk at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:79
run_work_thunk at D:\buildbot\worker\package_win64\build\usr\share\juld\src\Distributed.jl:75
 [4] #invokelatest#1 at .\essentials.jl:710 [inlined]
 [5] invokelatest at .\essentials.jl:709 [inlined]
 [6] require(::Base.PkgId) at .\loading.jl:931
 [7] require(::Module, ::Symbol) at .\loading.jl:923
 [8] top-level scope at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\macros.jl:200

The worker fails because the package JuMP is not available in its environment. Not really surprising since from the documentation it is clear that the workers do not inherit the global state of the master process (including the environment).

help?> addprocs()
  addprocs(; kwargs...) -> List of process identifiers

  Equivalent to addprocs(Sys.CPU_THREADS; kwargs...)

  Note that workers do not run a .julia/config/startup.jl startup     
  script, nor do they synchronize their global state (such as global  
  variables, new method definitions, and loaded modules) with any of  
  the other running processes.

But if we want to activate an environment in the worker as well…

julia> @everywhere begin
           import Pkg
           Pkg.activate()
       end
 Activating environment at `c:\Users\Martin Cornejo\MyProject\Project.toml`
      From worker 2:     Activating environment at `C:\Users\Martin Cornejo\.julia\environments\v1.5\Project.toml`

… instead of activating the specific project environment, it remains on the main environment (v1.5). Even though the process is aware of the current directory (where there are Project.toml and Manifest.toml files located)

julia> @everywhere 2 pwd()
"C:\\Users\\Martin Cornejo\\MyProject"

I understand that the procs would not be generally created in the same machine. But somehow it should be possible to specify the environment for these.

oheil · August 12, 2020, 1:35pm

I am not sure if this is the problem, but this is working for me:


julia> addprocs()

julia> @everywhere using Pkg

(@v1.5) pkg> activate PlayGround
 Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`

julia> @everywhere Pkg.activate("PlayGround")
 Activating new environment at `C:\WINDOWS\system32\PlayGround\Project.toml`
      From worker 5:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 4:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 8:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 9:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 3:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 6:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 7:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 2:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`

julia> using PlayGround

julia> @everywhere using PlayGround

The first using PlayGround is doing the precompilation. When I do just @everywhere using PlayGround I get an error, that the cache file can’t be written. The worker probably all do the precompilation concurrently which clearly has to fail. So first on the master for precompile and afterwards distribute it.

(EDIT: missing line @everywhere using Pkg)

martincornejo · August 12, 2020, 1:53pm

Is this not exactly what I was attempting above?

From my understanding, the @everywhere macro does exactly that when using a not-yet-loaded package

oheil · August 12, 2020, 2:11pm

Definitely not exactly the same as you can see. I am not sure what your begin… import…activate…end does and if it should be exactly the same as what I do. Thats why I said, I am not sure. But I thought, the slightly different way how I do it may be of some help for you, because it seems to work:

Example: The package GeneralizedGenerated is only available in my environment PlayGround:

julia> Pkg.activate()
 Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`

julia> using GeneralizedGenerated
ERROR: ArgumentError: Package GeneralizedGenerated not found in current path:
- Run `import Pkg; Pkg.add("GeneralizedGenerated")` to install the GeneralizedGenerated package.

Stacktrace:
 [1] require(::Module, ::Symbol) at .\loading.jl:893

julia> @everywhere Pkg.activate()
 Activating       From worker 7:         Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`

      From worker 2:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`
      From worker 8:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`
      From worker 3:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`
      From worker 9:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`
      From worker 5:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`
      From worker 4:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`
      From worker 6:     Activating environment at `C:\Users\USER\.julia\environments\v1.5\Project.toml`

julia> @everywhere using GeneralizedGenerated
ERROR: ArgumentError: Package GeneralizedGenerated not found in current path:
- Run `import Pkg; Pkg.add("GeneralizedGenerated")` to install the GeneralizedGenerated package.

Stacktrace:
 [1] require(::Module, ::Symbol) at .\loading.jl:893
 [2] top-level scope at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\macros.jl:200

julia> Pkg.activate("PlayGround")
 Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`

julia> using GeneralizedGenerated

julia> @everywhere Pkg.activate("PlayGround")
 Activating       From worker 8:         Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`

      From worker 7:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 3:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 5:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 9:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 2:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 6:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`
      From worker 4:     Activating environment at `C:\Users\USER\.julia\dev\PlayGround\Project.toml`

julia> @everywhere using GeneralizedGenerated

So no error at all if PlayGround is activated on master and worker.

Apparently not, because I got an error. Do you have a link to something e.g. in the docs, stating, that @everywhere using ... assures that the package is precompiled at first?

oheil · August 12, 2020, 2:15pm

This may be important for your attempt:

(PlayGround) pkg> ?activate
  activate
  activate [--shared|--temp] [path]

  Activate the environment at the given path, or the home project environment if no path is specified. The active environment is the
  environment that is modified by executing package commands. When the option --shared is given, path will be assumed to be a directory
  name and searched for in the environments folders of the depots in the depot stack. In case no such environment exists in any of the
  depots, it will be placed in the first depot of the stack. Use the temp option to create temporary environments. This should be useful
  for experimenting with packages.

Highlighted:
or the home project environment if no path is specified

home project is not the current path.

martincornejo · August 12, 2020, 3:15pm

That’s it, thanks! Specifying the path is important, the following works:

julia> @everywhere begin
          import Pkg
          Pkg.activate(@__DIR__)
       end