[ANN] PackageCompiler with incremental system images

compilation
precompilation

#1

I just tagged a new version of PackageCompiler that supports incremental compilation of system images and 1.0 compatible snooping of precompile statements.
This allows to to ahead of time compile your packages and remove JIT overhead!
There is some documentation in the README and I wrote some short instructions how one can compile a package and turn it into a docker image with nextjournal.

You can also try some images online by signing up (signup code julia1.0) and remixing these articles:

dataframes + query

plots & gr backend

makie & opengl backend

makie & cairo backend

If you find any package missing, anyone can create a new image with the above instructions (and maybe post it here?).

I must admit, this release is not 100% finished, since it took much more time and frustration to fix every single way things can go wrong when loading packages + precompiling them, leaving less time for polish…
com-gif-maker

Anyways, I put up a little list in the initial PR of what I think still needs to be done. Any help would be very much appreciated :slight_smile:

There is also a rename to ProjectCompiler pending.
I’m aiming to do the rename for the next proper release, so if anyone has a much better name, feel free to comment in that thread.

Best,
Simon


#2

Wow, that is awesome!!


#3

Sick. Thank you!!


#4

This is amazing, and i just tried compiling Plots,DataFrames,CSV togheter by following your guidelines on Nextjournal, actually i copied the exact commands on that journal, and the procedure worked, the only problem now is that julia starts with this warning. Any idea on how could i fix this?

francesco$ julia
┌ Warning: Error requiring IteratorInterfaceExtensions from Tables:
│ cannot assign variable IteratorInterfaceExtensions.IteratorInterfaceExtensions from module Tables
│ Stacktrace:
│ [1] top-level scope at none:0
│ [2] (::getfield(Tables, Symbol("##12#24")))() at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [3] err(::getfield(Tables, Symbol("##12#24")), ::Module, ::String) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [4] withpath(::getfield(Tables, Symbol("##11#23")), ::String) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [5] listenpkg(::getfield(Tables, Symbol("##10#22")), ::Base.PkgId) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [6] init() at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
└ @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40


#5

Hm, that must be a Tables error!? Maybe bad interaction with Requires ?


#6

Unfortunately i’m a total newbie, i started using and loving Julia, just a week ago…
Looking at the error i see that your analysis makes total sense, perhaps Requires wants some object that has been skipped by the the testunits and is not in the sysimg, or perhaps i’m completely on the wrong path…

The only downside to Julia are it’s precompilation times, and PackageCompiler can fix this, so i first precompiled Plots alone and it worked great! then i tried to precompile those other packages but the problem described above appeared.
By the way,

using DataFrames, CSV, StatsPlots
import ForwardDiff

This is what i’m using in my first Julia project, killing JIT times for all of them would have been marvelous!
For now i reinstalled the julia binary in order to revert to the original sysimg. But as soon as a solution comes up i’ll be using Pkg.Compiler again, it is really awesome.


#7

The problem might be that there is a @require IteratorInterfaceExtensions clause here, even though IteratorInterfaceExtensions is just a REQUIRED package of Tables.jl and also just normally used here in Tables.jl. Most likely the code in that @require clause should just be moved out of the @require clause, I don’t see a reason why it would be behind a @require clause.


#8

Unfortunately not working for Gtk.jl:

ERROR: LoadError: LoadError: LoadError: UndefVarError: BINDIR not defined
Stacktrace:
 [1] include at ./boot.jl:326 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1038
 [3] _require(::Base.PkgId) at ./loading.jl:986
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
 [6] include at ./boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1038
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] top-level scope at none:0
in expression starting at /Users/knopp/.julia/packages/WinRPM/Y9QdZ/src/WinRPM.jl:6

#9

Same goes for HDF5.


#10

it seems that the general repository is capping the bindeps version for these packages, which in turn caps the compat version, preventing Compat.Sys.BINDIR from working on windows


quick workaround for windows users until this gets sorted: do a dev BinDeps and replace Compat.Sys.BINDIR with Sys.BINDIR before running packagecompiler


#11

This looks really great, thanks!

I tried compiling Measurements.jl, but I get the error

 ErrorException("Task cannot be serialized")

like the one reported at https://github.com/JuliaLang/PackageCompiler.jl/issues/148. In Measurements.jl there is indeed a global constant: https://github.com/JuliaPhysics/Measurements.jl/blob/6f04d0f63bccb9d74ca680e9461dc0d41327f0d1/src/Measurements.jl#L68 But even if I change its initialization to Uint64[] PackageCompiler fails to compile it, with the same error. Is there another way to initialize the constant and make PackageCompiler happy?


#12

thanks, and what is the proper fix? The issue is not just Windows by the way, I am on OSX and get that error.


#13

I really don’t know. I’m not associated with any of these packages or the methodology used to decide compatibility in the general repo. Perhaps there’s no general solution for packages to work with PackageCompiler without the package maintainers explicitly testing


#14

btw, I confused myself with the general registry compat.toml above. I had old versions of compat on my machine, but after cleaning out my user directory and installing again, I get the BINDIR error even though only recent Compat versions are installed.

removing references in WinRPM and BinDeps to Compat.Sys.BINDIR still gets rid of the BINDIR not defined for me, but I think this is because Compat.Sys.BINDIR is defined in the __init__() function of Compat.Sys which I guess would not be available during compilation.

Along with the “Task cannot be serialized” error mentioned above, it seems that in order to have PackageCompiler work, we’d need to

  1. remove references to BINDIR in WinRPM and BinDeps
  2. have every package that we want precompiled put const global references inside the __init__ function

or maybe PackageCompiler could be updated with some magic to itself automatically modify any julia source being compiled to move const globals to __init__. Does that make any sense?


#15

See the discussion within Pkg issue.


The robust way to have binary dependencies work with AOT + deployment strategy is not to have to deal with lib paths at all - let Julia (Pkg) do it for you and just give you a “handle” to the lib based on som UUID of the binary artifact.
Untill than, only hacking package after package might help.


#16

I guess one will need to update bindeps and winrpm;)
it would also be helpful, to filter out packages that are known to be not needed in the system image.


#17

I am really looking forward to using this, once it is working with more packages.


#18

I’m looking forward to doing this to ALL my installed packages :heart_eyes:


#19

Thank you @sdanisch for your work! It is amazing what you are doing for this community! :heart:

Looking forward to deploying packages without JIT overhead!


#20

With package compiler, what happen when I need to update a compiled package? Is it as easy as recompiling after update? Will old versions of packages be stuck in the sysimage forevermore?