[ANN] PackageCompiler.jl

PackageCompiler.jl will make it easier to ahead of time compile your package and remove the JIT overhead by generating a custom user image.

Blog article explaining some details.

Let me know if you run into any issues via opening an issue!

Best,
Simon

36 Likes

Hey Simon, this looks really good. I will test.

One question: What if force=false? This will not override the system image? But how do a invoke julia to use the newly created system image.

More general: What is if changes to a package have been made? Can we detect this and at least give a warning?

Best,

Tobi

3 Likes

One question: What if force=false? This will not override the system image? But how do a invoke julia to use the newly created system image.

You will get a info message with the concrete path printed. E.g. something like julia -J path/image/...

More general: What is if changes to a package have been made? Can we detect this and at least give a warning?

That’s a good suggestion. Not sure on what level to resolve this - @tim.holy, is there infrastructure in Revise.jl that we could use for that?

Most likely, yes. The best model is probably how Revise.track(Base) works, where (on 0.7) any change since you last built julia gets detected and evaled appropriately. This depends on having a source-cache available, which julia now builds at the end of its build process (stored in $JULIA/usr/share/julia/base.cache).

Packages (on 0.7) work similarly, they now store a source-cache in the *.ji file so that Revise can compare current code against what it was when the package was precompiled.

So if you can create a source-cache, it should be pretty straightforward to load it into Revise and then to look for changes to the source files that occurred after the build.

Do either of you know if it is possible to build the system image in chunks. I.e. that it is possible to drop parts? This would be great since it would be possible to invalidate only those parts that were changed on disk.

I haven’t looked into that yet and I’m not the right one to ask. My assumption is, that if it would be easy, we’d have it by now :stuck_out_tongue:
It would make it pretty trivial to have e.g. julia/v0.x/lib/Package.so, and be over with this whole thing!

I hope I’m wrong!

This will have to work a little differently since modules X and Y may trigger the same code generation (e.g. of some Base method). But nevertheless this should be doable.

What’s the relationship between Transpiler.jl and PackageCompiler.jl?

There is almost none. PackageCompiler is about generating standalone executables + shared libraries from a Julia library, while Transpiler translates Julia code to OpenCL/OpenGL

2 Likes

Thanks for the clarification and please pardon my naivete on the subject at large (my questions probably should be in First Steps). Reading through the Github documentation and the referenced Julia Computing blog posts relating to AOT compilation, am I correct in assuming there is some relationship between PackageCompileler.jl and llvm-cbe.jl?