Why does Pkg.activate(".") require compilation? Time to first project activation

mkitti · July 5, 2023, 2:34am

Pkg.activate(".") seems to to take about 200 milliseconds, mostly due to compilation. It still takes 40 milliseconds even after activating the same environment again.

$ julia -e 'using Pkg; for i in 1:5; @time Pkg.activate("."); end'
  Activating project at `~/.julia/dev/ScriptUtils`
  0.220025 seconds (232.01 k allocations: 16.980 MiB, 98.52% compilation time)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.042125 seconds (81.89 k allocations: 5.737 MiB, 95.17% compilation time)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.001953 seconds (3.73 k allocations: 414.523 KiB)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.001633 seconds (3.73 k allocations: 414.180 KiB)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.001591 seconds (3.73 k allocations: 414.086 KiB)

I thought perhaps the second round of compilation had to due with the environment changing. Even if I initialize Julia with the correct environment these compilation times persist.

$ julia --project=. -e 'using Pkg; for i in 1:5; @time Pkg.activate("."); end'
  Activating project at `~/.julia/dev/ScriptUtils`
  0.219066 seconds (231.95 k allocations: 16.975 MiB, 98.48% compilation time)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.051085 seconds (81.89 k allocations: 5.737 MiB, 95.46% compilation time)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.001949 seconds (3.73 k allocations: 414.492 KiB)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.001591 seconds (3.73 k allocations: 414.148 KiB)
  Activating project at `~/.julia/dev/ScriptUtils`
  0.001618 seconds (3.73 k allocations: 414.055 KiB)

If I want to simplify usage of Julia code from the command line via an executable script, a Pkg.activate call is usually one of the first tasks.

Are we not precompiling Pkg.activate(::String) into the system image?

mkitti · July 5, 2023, 2:45am

It appears we are indeed not precompiling Pkg.activate at all.

github.com

JuliaLang/Pkg.jl/blob/2cd30ad9203e1b331b84387580304dd86492e936/src/Pkg.jl#L867-L885


      
          const precompile_script = """
              import Pkg
              _pwd = pwd()
              Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
              tmp = Pkg._run_precompilation_script_setup()
              $CTRL_C
              Pkg.add("TestPkg")
              Pkg.develop(Pkg.PackageSpec(path="TestPkg.jl"))
              Pkg.add(Pkg.PackageSpec(path="TestPkg.jl/"))
              Pkg.REPLMode.try_prompt_pkg_add(Symbol[:notapackage])
              Pkg.update(; update_registry=false)
              Pkg.precompile()
              ] add Te\t\t$CTRL_C
              ] st
              $CTRL_C
              Pkg._run_precompilation_script_artifact()
              rm(tmp; recursive=true)
              cd(_pwd)
              """

Edit: There is actually one precompilation of Pkg.activate, but it is for an empty Project.toml.

github.com

JuliaLang/Pkg.jl/blob/2cd30ad9203e1b331b84387580304dd86492e936/src/Pkg.jl#L800-L801


      
          touch("Project.toml")
          Pkg.activate(".")

Activating an empty Project.toml is fast, but perhaps this is not a useful case.

$ julia -e 'using Pkg; touch("Project.toml"); @time Pkg.activate(".")'
  Activating project at `~/src/Pkg.jl/test/blah`
  0.000665 seconds (525 allocations: 51.023 KiB)

mkitti · July 5, 2023, 3:00am

I submitted a pull request.

github.com/JuliaLang/Pkg.jl

Precompile Pkg.activate()

JuliaLang:master ← mkitti:mkitti/precompile_activate

opened 02:59AM - 05 Jul 23 UTC

mkitti

+4 -0

`Pkg.activate` is a necessary first step for running executable Julia scripts. F…or a novice Julia user, it is often easier to provide them with a script to run simply via `julia script.jl`. script.jl may contain the following contents. ``` #!/bin/env julia using Pkg Pkg.activate("/path/to/the/right/environment") using PackageInEnvironment ... ``` While trying to build such scripts, I found that `Pkg.activate` is contributing some latency. Activating the environment at https://github.com/mkitti/ScriptUtils.jl/tree/main/examples/Echo.jl can take 100 to 200 milliseconds on my computer, with most of the time being compilation. Even a second activation requires an additional 40 mlliseconds of compilation time. ``` $ julia --project=. -e 'using Pkg; for i in 1:5; @time Pkg.activate("."); end' Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl` 0.106829 seconds (159.75 k allocations: 11.833 MiB, 97.05% compilation time) Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl` 0.042188 seconds (81.92 k allocations: 5.750 MiB, 95.15% compilation time) Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl` 0.001958 seconds (3.76 k allocations: 419.430 KiB) Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl` 0.001606 seconds (3.76 k allocations: 419.180 KiB) Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl` 0.001637 seconds (3.76 k allocations: 419.133 KiB) ``` Alternating between the default environment and the current directory, also seems to involve compilation up to four activations. ``` $ julia --project=. -e 'using Pkg; for i in 1:5; @time Pkg.activate("."); @time Pkg.activate(); end' Activating project at `~/src/Pkg.jl` 0.115365 seconds (73.03 k allocations: 5.318 MiB, 99.10% compilation time) Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.9` 0.124273 seconds (166.67 k allocations: 12.133 MiB, 97.72% compilation time) Activating project at `~/src/Pkg.jl` 0.004956 seconds (742 allocations: 76.609 KiB, 88.74% compilation time) Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.9` 0.038794 seconds (81.91 k allocations: 5.739 MiB, 94.59% compilation time) Activating project at `~/src/Pkg.jl` 0.000546 seconds (727 allocations: 75.516 KiB) Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.9` 0.001853 seconds (3.77 k allocations: 417.180 KiB) Activating project at `~/src/Pkg.jl` 0.000377 seconds (728 allocations: 75.812 KiB) Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.9` 0.001656 seconds (3.77 k allocations: 417.273 KiB) Activating project at `~/src/Pkg.jl` 0.000365 seconds (727 allocations: 75.516 KiB) Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.9` 0.001650 seconds (3.77 k allocations: 417.180 KiB) ``` As such I propose adding four `Pkg.activate` calls to the Pkg.jl precompilation script, alternating between the default environment and TestPkg.jl.

Edit: Doing this with a local build seems to have helped with the default environment, but there is still some compilation needed to activate one of my test enviornments.

$ ./julia -e 'using Pkg; for i in 1:5; @time Pkg.activate(); end' --trace-compile=stderr;
  Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.11`
  0.000518 seconds (543 allocations: 41.523 KiB)
  Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.11`
  0.000251 seconds (538 allocations: 41.133 KiB)
  Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.11`
  0.000215 seconds (538 allocations: 41.133 KiB)
  Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.11`
  0.000213 seconds (538 allocations: 41.133 KiB)
  Activating project at `/tmp/tmp.298dOvl2Nq/environments/v1.11`
  0.000205 seconds (539 allocations: 41.430 KiB)

$ ./julia -e 'using Pkg; for i in 1:5; @time Pkg.activate("/home/mkitti/.julia/dev/ScriptUtils/examples/Echo.jl"); end' --trace-compile=stderr;
  Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl`
precompile(Tuple{Type{Base.Generator{I, F} where F where I}, Pkg.Types.var"#52#55"{String, String}, Array{Any, 1}})
precompile(Tuple{typeof(Base.collect_similar), Array{Any, 1}, Base.Generator{Array{Any, 1}, Pkg.Types.var"#52#55"{String, String}}})
precompile(Tuple{Pkg.Types.var"#52#55"{String, String}, Base.Dict{String, Any}})
precompile(Tuple{Type{Array{Dates.DateTime, 1}}, UndefInitializer, Tuple{Int64}})
precompile(Tuple{typeof(Base.collect_to_with_first!), Array{Dates.DateTime, 1}, Dates.DateTime, Base.Generator{Array{Any, 1}, Pkg.Types.var"#52#55"{String, String}}, Int64})
precompile(Tuple{typeof(Core.Compiler.eltype), Type{Array{UInt64, 1}}})
precompile(Tuple{typeof(Base.deepcopy_internal), Tuple{UInt64}, Base.IdDict{Any, Any}})
precompile(Tuple{typeof(Base.Ryu.writefixed), Float64, Int64})
  0.110661 seconds (134.34 k allocations: 10.412 MiB, 96.54% compilation time)
  Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl`
precompile(Tuple{typeof(Base.ident_cmp), Tuple{UInt64}, Tuple{UInt64}})
  0.009558 seconds (9.16 k allocations: 830.914 KiB, 77.94% compilation time)
  Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl`
  0.001870 seconds (3.44 k allocations: 406.727 KiB)
  Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl`
  0.001649 seconds (3.44 k allocations: 406.727 KiB)
  Activating project at `~/.julia/dev/ScriptUtils/examples/Echo.jl`
  0.001662 seconds (3.44 k allocations: 407.023 KiB)