Precompilation after Julia Update

Tooling
1

Hi,

updating Julia versions can be annoying because precompilation for big projects take quite long.

I thought it’s clever to do the update, and then run a script which activates all my dev folders and instantiates them. That should trigger precompilation for many packages. Especially on my Laptop this seems to help a lot:

Just run this function, takes some minutes but then most of the stuff should be precompiled.

Old code, see updated below 
using Pkg 

function precompile_dev_folders()
    cd("/home/fxw/.julia/dev")
    for folder in readdir()
        if isdir(folder)
            println("In folder: $folder")
            cd(folder)
            try
                Pkg.activate(".")
                Pkg.instantiate()
                try
                    cd("examples")
                    Pkg.activate(".")
                    Pkg.instantiate()
                    cd("..")
                catch
                    println("No examples in $folder")
                end
            catch e
                println("Error in folder: $folder")
                println(e)
            end
            cd("..")
        end
    end
end
6 Likes
2

This seems particularly focused on developers.

I think you could adapt this to go through recent Manifest.toml usage by using ~/.julia/logs/manifest_usage.toml

1 Like
3

Thanks for the tip!

using TOML
using Dates
using Pkg

"""
    precompile_envs(fname=".julia/logs/manifest_usage.toml"; 
                    time_diff=Dates.CompoundPeriod(Dates.Day(30)))

Precompile the environments listed in the TOML file `fname` if the last time they were used is greater than `time_diff`.
`fname` is the path to the TOML file that contains the information about the last time an environment on your machine was used.
As default `time_diff` is set to 30 days. 

So this script precompiles the environments that were used in the last 30 days. 
This is especially useful to save time after updating Julia.

"""
function precompile_envs(fname=".julia/logs/manifest_usage.toml"; time_diff=Dates.CompoundPeriod(Dates.Day(30)))
    # Load the project file
    toml = TOML.parsefile(fname)
    now = Dates.now()

    # Loop over the keys in the TOML file
    for key in keys(toml)

        for minor in deleteat!(collect(1:50), VERSION.minor)
            if occursin("/v1.$(minor)/", key)
                continue
            end
        end
        time = toml[key][1]["time"]

        try
            # if time difference greater than time_diff, precompile
            if now - time < time_diff
                # Precompile the project if possible and catch errors
                try
                    @info "Precompiling $key"
                    Pkg.activate(dirname(key))
                    Pkg.precompile()
                catch e
                end
                # deactivate the project
                Pkg.activate()
            end
        catch e
            if isa(e, InterruptException)
                @info "Interrupted"
                return
            else
                rethrow(e)
            end
        end
    end
end
7 Likes
4

It would probably be useful to exclude the default environments of earlier Julia versions from the script.

2 Likes
5

Yes, make sense!
I think those lines should do it, I added it above.

3 Likes
6

Great tool! There’s a little typo in the function or the docs, as default time_diff doesn’t match.

I’ve opened an issue at Pkg.jl long time ago for exactly this purpose, though it didn’t get attention and I haven’t reiterated it since. We could ask the devs if your implementation would be welcomed in the code.

I’ll definitely use it, thank you!

2 Likes
7

It seems like this also precompiles environments with a different julia version than the currently active one:

julia> precompile_envs()
[ Info: Precompiling /home/dennishb/.julia/dev/Peaks.jl/Manifest.toml
  Activating project at `~/.julia/dev/Peaks.jl`
  Activating project at `~/.julia/environments/v1.10`
[ Info: Precompiling /home/dennishb/.vscode/extensions/julialang.language-julia-1.79.2/scripts/environments/languageserver/v1.10/Manifest.toml
  Activating project at `~/.vscode/extensions/julialang.language-julia-1.79.2/scripts/environments/languageserver/v1.10`
  Activating project at `~/.julia/environments/v1.10`
[ Info: Precompiling /home/dennishb/.julia/environments/v1.11/Manifest.toml
  Activating project at `~/.julia/environments/v1.11`
┌ Warning: The active manifest file has dependencies that were resolved with a different julia version (1.11.0-beta2). Unexpected behavior may occur.
└ @ ~/.julia/environments/v1.11/Manifest.toml:0
  Activating project at `~/.julia/environments/v1.10`
[ Info: Precompiling /home/dennishb/.julia/environments/v1.10/Manifest.toml
  Activating project at `~/.julia/environments/v1.10`
  Activating project at `~/.julia/environments/v1.10`
8

Are you using 1.10.4 and it compiled the .julia/environments/v1.11?

I think the for loop only skips versions below the current version but not higher.
I just did a quick update.

9

I first ran it with 1.10.4, then 1.11 beta, then did Pkg.update(), and then 1.10.4 again. The warning appears to have come because the environment was resolved when in the 1.11 channel (juliaup), which ended up in printing a warning when I went back to a previous julia version (1.10.4).

So I really abused it jumping julia version like that. But I was just thinking that a simple version check could make it robust also against such abuse. Otherwise, the function worked great, so thank you for posting it :grin:

1 Like
10

Now it is registered as PrecompileAfterUpdate.jl

3 Likes