How to programmatically update my PackageCompiler's sysimage, using a precompile_execution_file?

General Usage
, , ,
1

Hi all!

I am using PackageCompiler to speed up Julia usage. It works enthusiastically well, and it saves me some time, but I also want to keep the sysimage updated. Specifically, I want to update it (even manually) once in a while, when:

  • a package gets updated
  • I use a new function in my scripts

First and main question (to avoid the XY problem): how can I do that?

I’ve thought I could embed the package compilation in a script to invoke periodically:

## update_sysimage.jl

using PackageCompiler;

PackageCompiler.create_sysimage(["Package1", "Package2"];
                                sysimage_path="/path/to/sysimage.so",
                                precompile_execution_file="???.jl")

Since I run multiple scripts in a specific directory, and since you have to use the precompile_execution_file to instruct Julia to detect and compile the functions you use, what should I put in there, to make it aware of new functions? I thought I can simply get the list of the scripts I wrote, and include them all:

# precompile_execution_file.jl
script_files = readdir(abspath(scripts_source_dir), join=true);

include.(script_files);

Now, I think this should work (I’m not sure), BUT:

  1. Do I do unnecessary computation? Is there a more efficient/idiomatic way?

  2. When using PackageCompiler.create_sysimage, does the precompile_execution_file effectively get run? IOW, will it be run, and modify my data with side-effects, or PackageCompiler simply reads it to extract the signatures of the functions to compile?

  3. do you know any method to automatically detect also the list of packages to pass in the first argument array?

Thanks!

2

You can use something like this to get a list of all packages added to the current environment:

julia> import Pkg
julia> packages = Symbol.(keys(Pkg.project().dependencies))
6-element Vector{Symbol}:
 :Revise
 :Debugger
 :BenchmarkTools
 :OhMyREPL
 :Infiltrator
 :ProfileView

Yes, the precompile_execution_file will effectively get run, and will potentially have side-effects. This is especially adequate if you have unit tests, because presumably they call a large fraction of your code without having any harmful side effect.

If you have a collection of scripts that actually modify data, then it may not be a very good idea to use it in the precompile_execution_file. Or maybe you can setup a sandboxed environment in which the scripts run and alter a copy of your data, but not the real thing?

Alternatively, you can provide a precompile_statements_file, in which PackageCompiler will simply read the list of methods to compile. But this list has to be generated ahead of time (e.g using julia --trace-compile or SnoopCompile), and I’m pretty sure at some point you’ll have to actually execute the code you want to compile. But at least you can generate the statements list once, and re-use it multiple times (knowing that if it gets out-of-date, things should still work but the benefits of precompilation may be reduced).

1 Like