# \[ANN\] PackageCompiler with incremental system images

**URL:** <https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489>\
**Category:** Community\
**Tags:** compilation, precompilation\
**Created:** [February 5, 2019, 11:21pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489 "2019-02-05T23:21:16Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [February 5, 2019, 11:21pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/1 "2019-02-05T23:21:16Z")

</div>

I just tagged a new version of [PackageCompiler](https://github.com/JuliaLang/PackageCompiler.jl) that supports incremental compilation of system images and 1.0 compatible snooping of precompile statements.  
This allows to to ahead of time compile your packages and remove JIT overhead!  
There is some documentation in the [README](https://github.com/JuliaLang/PackageCompiler.jl/blob/master/README.md) and I wrote some [short instructions](https://nextjournal.com/sdanisch/static-cairomakie) how one can compile a package and turn it into a docker image with [nextjournal](https://nextjournal.com/).

You can also try some images online by signing up (signup code `julia1.0`) and remixing these articles:

[dataframes + query](https://nextjournal.com/sdanisch/data-remix)

[plots & gr backend](https://nextjournal.com/sdanisch/plots-remix)

[makie & opengl backend](https://nextjournal.com/sdanisch/glmakie-remix)

[makie & cairo backend](https://nextjournal.com/sdanisch/cairomakie-remix)

If you find any package missing, anyone can create a new image with the above instructions (and maybe post it here?).

I must admit, this release is not 100% finished, since it took much more time and frustration to fix every single way things can go wrong when loading packages + precompiling them, leaving less time for polish…  
 ![com-gif-maker](https://global.discourse-cdn.com/julialang/original/3X/e/4/e4f4ffc821755233afbf585b90f5876f3b63b764.gif)

Anyways, I put up a little list in the initial PR of what I think [still needs to be done](https://github.com/JuliaLang/PackageCompiler.jl/pull/158#issuecomment-458564317). Any help would be very much appreciated 🙂

There is also a [rename to ProjectCompiler pending](https://github.com/JuliaLang/PackageCompiler.jl/issues/160).  
I’m aiming to do the rename for the next proper release, so if anyone has a much better name, feel free to comment in that thread.

Best,  
Simon

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 6, 2019, 1:01am UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/2 "2019-02-06T01:01:42Z")

</div>

Wow, that is awesome!!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [February 6, 2019, 8:16am UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/3 "2019-02-06T08:16:57Z")

</div>

Sick. Thank you!!

---

<div class="post-metadata">

**Author:** ![francesco.alemanno](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@francesco.alemanno](https://discourse.julialang.org/u/francesco.alemanno)\
**Post date:** [February 6, 2019, 3:48pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/4 "2019-02-06T15:48:15Z")

</div>

This is amazing, and i just tried compiling Plots,DataFrames,CSV togheter by following your guidelines on Nextjournal, actually i copied the exact commands on that journal, and the procedure worked, the only problem now is that julia starts with this warning. Any idea on how could i fix this?

> francesco$ julia  
> ┌ Warning: Error requiring IteratorInterfaceExtensions from Tables:  
> │ cannot assign variable IteratorInterfaceExtensions.IteratorInterfaceExtensions from module Tables  
> │ Stacktrace:  
> │ [1] top-level scope at none:0  
> │ [2] (::getfield(Tables, Symbol(“##12#24”)))() at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?  
> │ [3] err(::getfield(Tables, Symbol(“##12#24”)), ::Module, ::String) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?  
> │ [4] withpath(::getfield(Tables, Symbol(“##11#23”)), ::String) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?  
> │ [5] listenpkg(::getfield(Tables, Symbol(“##10#22”)), ::Base.PkgId) at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?  
> │ [6] **init** () at /Applications/Julia-1.1.app/Contents/Resources/julia/lib/julia/sys.dylib:?  
> └ @ Requires ~/.julia/packages/Requires/9Jse8/src/require.jl:40

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [February 6, 2019, 3:56pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/5 "2019-02-06T15:56:35Z")

</div>

Hm, that must be a Tables error!? Maybe bad interaction with `Requires` ?

---

<div class="post-metadata">

**Author:** ![francesco.alemanno](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@francesco.alemanno](https://discourse.julialang.org/u/francesco.alemanno)\
**Post date:** [February 6, 2019, 4:08pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/6 "2019-02-06T16:08:40Z")

</div>

Unfortunately i’m a total newbie, i started using and loving Julia, just a week ago…  
Looking at the error i see that your analysis makes total sense, perhaps Requires wants some object that has been skipped by the the testunits and is not in the sysimg, or perhaps i’m completely on the wrong path…

The only downside to Julia are it’s precompilation times, and PackageCompiler can fix this, so i first precompiled Plots alone and it worked great! then i tried to precompile those other packages but the problem described above appeared.  
By the way,

> using DataFrames, CSV, StatsPlots  
> import ForwardDiff

This is what i’m using in my first Julia project, killing JIT times for all of them would have been marvelous!  
For now i reinstalled the julia binary in order to revert to the original sysimg. But as soon as a solution comes up i’ll be using Pkg.Compiler again, it is really awesome.

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [February 6, 2019, 5:13pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/7 "2019-02-06T17:13:25Z")

</div>

The problem might be that there is a `@require IteratorInterfaceExtensions` clause [here](https://github.com/JuliaData/Tables.jl/blob/265ff4e59f060ec67f6be4cd9630c124a9c07df9/src/Tables.jl#L29), even though `IteratorInterfaceExtensions` is just a `REQUIRED` package of Tables.jl and also just normally used [here](https://github.com/JuliaData/Tables.jl/blob/265ff4e59f060ec67f6be4cd9630c124a9c07df9/src/Tables.jl#L5) in Tables.jl. Most likely the code in that `@require` clause should just be moved out of the `@require` clause, I don’t see a reason why it would be behind a `@require` clause.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [February 6, 2019, 5:23pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/8 "2019-02-06T17:23:19Z")

</div>

Unfortunately not working for Gtk.jl:

```julia
ERROR: LoadError: LoadError: LoadError: UndefVarError: BINDIR not defined
Stacktrace:
 [1] include at ./boot.jl:326 [inlined]
 [2] include_relative(::Module, ::String) at ./loading.jl:1038
 [3] _require(::Base.PkgId) at ./loading.jl:986
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
 [6] include at ./boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1038
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] top-level scope at none:0
in expression starting at /Users/knopp/.julia/packages/WinRPM/Y9QdZ/src/WinRPM.jl:6

```

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [February 6, 2019, 5:31pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/9 "2019-02-06T17:31:52Z")

</div>

Same goes for HDF5.

---

<div class="post-metadata">

**Author:** ![alandion](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alandion/32/5591_2.png) [@alandion](https://discourse.julialang.org/u/alandion)\
**Post date:** [February 6, 2019, 7:07pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/10 "2019-02-06T19:07:15Z")

</div>

it seems that the general repository is capping the bindeps version for these packages, which in turn caps the compat version, preventing Compat.Sys.BINDIR from working on windows

> <https://github.com/JuliaRegistries/General/blob/master/G/Gtk/Compat.toml>

> <https://github.com/JuliaRegistries/General/blob/master/H/HDF5/Compat.toml>

quick workaround for windows users until this gets sorted: do a `dev BinDeps` and replace `Compat.Sys.BINDIR` with `Sys.BINDIR` before running packagecompiler

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [February 6, 2019, 8:37pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/11 "2019-02-06T20:37:26Z")

</div>

This looks really great, thanks!

I tried compiling `Measurements.jl`, but I get the error

```julia
 ErrorException("Task cannot be serialized")

```

like the one reported at [https://github.com/JuliaLang/PackageCompiler.jl/issues/148](https://github.com/JuliaLang/PackageCompiler.jl/issues/148). In `Measurements.jl` there is indeed a global constant: [https://github.com/JuliaPhysics/Measurements.jl/blob/6f04d0f63bccb9d74ca680e9461dc0d41327f0d1/src/Measurements.jl#L68](https://github.com/JuliaPhysics/Measurements.jl/blob/6f04d0f63bccb9d74ca680e9461dc0d41327f0d1/src/Measurements.jl#L68) But even if I change its initialization to `Uint64[]` `PackageCompiler` fails to compile it, with the same error. Is there another way to initialize the constant and make `PackageCompiler` happy?

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [February 6, 2019, 10:08pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/12 "2019-02-06T22:08:51Z")

</div>

thanks, and what is the proper fix? The issue is not just Windows by the way, I am on OSX and get that error.

---

<div class="post-metadata">

**Author:** ![alandion](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alandion/32/5591_2.png) [@alandion](https://discourse.julialang.org/u/alandion)\
**Post date:** [February 6, 2019, 10:22pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/13 "2019-02-06T22:22:42Z")

</div>

I really don’t know. I’m not associated with any of these packages or the methodology used to decide compatibility in the general repo. Perhaps there’s no general solution for packages to work with PackageCompiler without the package maintainers explicitly testing

---

<div class="post-metadata">

**Author:** ![alandion](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alandion/32/5591_2.png) [@alandion](https://discourse.julialang.org/u/alandion)\
**Post date:** [February 7, 2019, 2:52am UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/14 "2019-02-07T02:52:51Z")

</div>

btw, I confused myself with the general registry compat.toml above. I had old versions of compat on my machine, but after cleaning out my user directory and installing again, I get the `BINDIR` error even though only recent Compat versions are installed.

removing references in `WinRPM` and `BinDeps` to `Compat.Sys.BINDIR` still gets rid of the `BINDIR not defined` for me, but I think this is because `Compat.Sys.BINDIR` is defined in the ` __init__ ()` function of `Compat.Sys` which I guess would not be available during compilation.

Along with the “Task cannot be serialized” error mentioned above, it seems that in order to have `PackageCompiler` work, we’d need to

1. remove references to `BINDIR` in `WinRPM` and `BinDeps`
2. have every package that we want precompiled put const global references inside the ` __init__ ` function

or maybe `PackageCompiler` could be updated with some magic to itself automatically modify any julia source being compiled to move const globals to ` __init__ `. Does that make any sense?

---

<div class="post-metadata">

**Author:** ![phlavenk](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/phlavenk/32/1873_2.png) [@phlavenk](https://discourse.julialang.org/u/phlavenk)\
**Post date:** [February 7, 2019, 5:49am UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/15 "2019-02-07T05:49:41Z")

</div>

See the discussion within Pkg issue.  
[https://github.com/JuliaLang/Pkg.jl/issues/841#issuecomment-459333186](https://github.com/JuliaLang/Pkg.jl/issues/841#issuecomment-459333186)  
The robust way to have binary dependencies work with AOT + deployment strategy is not to have to deal with lib paths at all - let Julia (Pkg) do it for you and just give you a “handle” to the lib based on som UUID of the binary artifact.  
Untill than, only hacking package after package might help.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [February 7, 2019, 7:29am UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/16 "2019-02-07T07:29:58Z")

</div>

I guess one will need to update bindeps and winrpm;)  
it would also be helpful, to filter out packages that are known to be not needed in the system image.

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [February 7, 2019, 9:21am UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/17 "2019-02-07T09:21:58Z")

</div>

I am really looking forward to using this, once it is working with more packages.

---

<div class="post-metadata">

**Author:** ![mkborregaard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkborregaard/32/556_2.png) [@mkborregaard](https://discourse.julialang.org/u/mkborregaard)\
**Post date:** [February 7, 2019, 12:07pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/18 "2019-02-07T12:07:13Z")

</div>

I’m looking forward to doing this to ALL my installed packages 😍

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [February 7, 2019, 12:21pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/19 "2019-02-07T12:21:12Z")

</div>

Thank you @sdanisch for your work! It is amazing what you are doing for this community! ❤

Looking forward to deploying packages without JIT overhead!

---

<div class="post-metadata">

**Author:** ![Mason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mason/32/2423_2.png) [@Mason](https://discourse.julialang.org/u/Mason)\
**Post date:** [February 7, 2019, 3:49pm UTC](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489/20 "2019-02-07T15:49:43Z")

</div>

With package compiler, what happen when I need to update a compiled package? Is it as easy as recompiling after update? Will old versions of packages be stuck in the sysimage forevermore?

[Next page](https://discourse.julialang.org/t/ann-packagecompiler-with-incremental-system-images/20489.md?page=2)