I just tagged a new version of PackageCompiler that supports incremental compilation of system images and 1.0 compatible snooping of precompile statements.
This allows to to ahead of time compile your packages and remove JIT overhead!
There is some documentation in the README and I wrote some short instructions how one can compile a package and turn it into a docker image with nextjournal.
You can also try some images online by signing up (signup code julia1.0) and remixing these articles:
If you find any package missing, anyone can create a new image with the above instructions (and maybe post it here?).
I must admit, this release is not 100% finished, since it took much more time and frustration to fix every single way things can go wrong when loading packages + precompiling them, leaving less time for polish…
Anyways, I put up a little list in the initial PR of what I think still needs to be done. Any help would be very much appreciated
There is also a rename to ProjectCompiler pending.
I’m aiming to do the rename for the next proper release, so if anyone has a much better name, feel free to comment in that thread.
This is amazing, and i just tried compiling Plots,DataFrames,CSV togheter by following your guidelines on Nextjournal, actually i copied the exact commands on that journal, and the procedure worked, the only problem now is that julia starts with this warning. Any idea on how could i fix this?
francesco$ julia
┌ Warning: Error requiring IteratorInterfaceExtensions from Tables:
│ cannot assign variable IteratorInterfaceExtensions.IteratorInterfaceExtensions from module Tables
│ Stacktrace:
│ [1] top-level scope at none:0
│ [2] (::getfield(Tables, Symbol(“##12#24”)))() at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [3] err(::getfield(Tables, Symbol(“##12#24”)), ::Module, ::String) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [4] withpath(::getfield(Tables, Symbol(“##11#23”)), ::String) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [5] listenpkg(::getfield(Tables, Symbol(“##10#22”)), ::Base.PkgId) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
│ [6] init() at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?
â”” @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40
Unfortunately i’m a total newbie, i started using and loving Julia, just a week ago…
Looking at the error i see that your analysis makes total sense, perhaps Requires wants some object that has been skipped by the the testunits and is not in the sysimg, or perhaps i’m completely on the wrong path…
The only downside to Julia are it’s precompilation times, and PackageCompiler can fix this, so i first precompiled Plots alone and it worked great! then i tried to precompile those other packages but the problem described above appeared.
By the way,
using DataFrames, CSV, StatsPlots
import ForwardDiff
This is what i’m using in my first Julia project, killing JIT times for all of them would have been marvelous!
For now i reinstalled the julia binary in order to revert to the original sysimg. But as soon as a solution comes up i’ll be using Pkg.Compiler again, it is really awesome.
The problem might be that there is a @require IteratorInterfaceExtensions clause here, even though IteratorInterfaceExtensions is just a REQUIRED package of Tables.jl and also just normally used here in Tables.jl. Most likely the code in that @require clause should just be moved out of the @require clause, I don’t see a reason why it would be behind a @require clause.
ERROR: LoadError: LoadError: LoadError: UndefVarError: BINDIR not defined
Stacktrace:
[1] include at ./boot.jl:326 [inlined]
[2] include_relative(::Module, ::String) at ./loading.jl:1038
[3] _require(::Base.PkgId) at ./loading.jl:986
[4] require(::Base.PkgId) at ./loading.jl:858
[5] require(::Module, ::Symbol) at ./loading.jl:853
[6] include at ./boot.jl:326 [inlined]
[7] include_relative(::Module, ::String) at ./loading.jl:1038
[8] include(::Module, ::String) at ./sysimg.jl:29
[9] top-level scope at none:0
in expression starting at /Users/knopp/.julia/packages/WinRPM/Y9QdZ/src/WinRPM.jl:6
it seems that the general repository is capping the bindeps version for these packages, which in turn caps the compat version, preventing Compat.Sys.BINDIR from working on windows
quick workaround for windows users until this gets sorted: do a dev BinDeps and replace Compat.Sys.BINDIR with Sys.BINDIR before running packagecompiler
I really don’t know. I’m not associated with any of these packages or the methodology used to decide compatibility in the general repo. Perhaps there’s no general solution for packages to work with PackageCompiler without the package maintainers explicitly testing
btw, I confused myself with the general registry compat.toml above. I had old versions of compat on my machine, but after cleaning out my user directory and installing again, I get the BINDIR error even though only recent Compat versions are installed.
removing references in WinRPM and BinDeps to Compat.Sys.BINDIR still gets rid of the BINDIR not defined for me, but I think this is because Compat.Sys.BINDIR is defined in the __init__() function of Compat.Sys which I guess would not be available during compilation.
Along with the “Task cannot be serialized” error mentioned above, it seems that in order to have PackageCompiler work, we’d need to
remove references to BINDIR in WinRPM and BinDeps
have every package that we want precompiled put const global references inside the __init__ function
or maybe PackageCompiler could be updated with some magic to itself automatically modify any julia source being compiled to move const globals to __init__. Does that make any sense?
See the discussion within Pkg issue. https://github.com/JuliaLang/Pkg.jl/issues/841#issuecomment-459333186
The robust way to have binary dependencies work with AOT + deployment strategy is not to have to deal with lib paths at all - let Julia (Pkg) do it for you and just give you a “handle” to the lib based on som UUID of the binary artifact.
Untill than, only hacking package after package might help.
I guess one will need to update bindeps and winrpm;)
it would also be helpful, to filter out packages that are known to be not needed in the system image.
With package compiler, what happen when I need to update a compiled package? Is it as easy as recompiling after update? Will old versions of packages be stuck in the sysimage forevermore?
