Pkg performance on Windows Dev Drive

Tooling Package Management
, , ,
1

I recently moved my Julia depot to a Dev Drive and am quite happy with the results, so I’m sharing my experience.

This contrived benchmark sped up from 176 to 8 seconds:

for i in 1:100_000
    open("file.txt", "w") do io
        println(io, "hello")
    end
end

To enable that I created a Dev Drive, and changed JULIA_DEPOT_PATH, JULIAUP_DEPOT_PATH and the TEMP & TMP environment variables to paths on the Dev Drive.

The standard Windows filesystem is slower than that of Linux for dealing with many small files. This tends to affect developers using package managers. With a Dev Drive you can achieve faster operations due to a combination of a faster filesystem (ReFS), and setting Windows Defender in performance mode such that it does asynchronous scanning. I’m not using Defender but another antivirus, and noticed its real time scanning slowing down or breaking workflows considerably. Luckily our ICT supported reconfiguring it for Dev Drives. So depending on your antivirus your speedups may be less.

Here is another benchmark measuring the time to install GLMakie in an empty depot. Even though download time will likely dominate this benchmark, it still sped up from 129 to 86 seconds for me.

$env:JULIA_DEPOT_PATH = ".julia-depot;"

# Pkg uses the temp dir, also move it to the Dev Drive
$env:TEMP = "Temp"
$env:TMP = "Temp"

# Disable precompilation so we can focus on IO performance
$env:JULIA_PKG_PRECOMPILE_AUTO = 0

julia --eval 'using Pkg; @time Pkg.add("GLMakie")'
# Dev Drive: 86.457165 seconds (18.75 M allocations: 1.929 GiB, 0.40% gc time, 6175 lock conflicts, 2.99% compilation time)
# Before:   129.727317 seconds (18.72 M allocations: 1.937 GiB, 0.28% gc time, 3334 lock conflicts, 1.93% compilation time)

In the docs Microsoft recommends redirecting TEMP and TMP to the Dev Drive. Though @ianshmean does Use temp dir on same filesystem for source install by IanButterworth · Pull Request #4180 · JuliaLang/Pkg.jl mean that Dev Drive users don’t need to set this from Julia 1.12 since Pkg will put temp files in the depot path as well?

2 Likes
2

That PR (and some tidy up PRs that followed) only affects a small amount of Pkg work during install.

If there’s motivation to do more like that then perhaps it could be widened

1 Like