How does Julia handle precompilation?

General Usage
1

I am trying to make an app using PackageCompiler.jl
I would like to use a precompile_statements_file.

My understanding is that I basically call

julia --trace-compile=precompile_statements.jl --project=@. precompile.jl

and get a file precompile_statements.jl with lots of precompile statements.
However, how can I make Julia compile all dependencies, the file seems too short?
I tried --compile=all but that seems not to be it. And I worry that there are some missed and not part of the system image.

2

A dumb way to do it is to clear the entire precompile cache, which is in [julia depot]/compiled/v1.X, where [julia depot] is the root of your depot (by default ~/.julia unless you changed it), and v1.X is the minor version you’re using.

It’s safe to delete the precompile cache, the only obvious consequence is that you’ll have to precompile all packages whose cache you delete, but you do want to do that (or at least for a fraction of the packages already precompiled in your depot)

3

My motivation is more in the direction of how to cache the precompilation of the PackageCompiler.jl .

My thinking is that if I have the precompile statements, for some changes this is enough and I do not need to run --trace-compile again and I can skip the precompile_execution_file step. The sysimage part of the app building process is taking long enough.

What happens with stuff that is missed, i.e. has no preccompile statement at runtime? Can this be compiled later? How does this work, when there is no source?

4

It’s not really clear what you’re describing here after talking about dependencies earlier. Do you mean call signatures that just happened to not execute in precompile.jl? For example, rand((true, false)) ? foo(1) : foo(1.0) end would randomly pick one call signature per run. In that case, the missed call signature can be JIT compiled later because PackageCompiler doesn’t trim anything from the bundled Julia runtime. The trim option of the experimental juliac does, so you’ll want to firmly specify everything you’ll call there like any other AOT-compiled language.

5

When I run trace-compile on a script some things get recorded by my feeling is that this is not everything that is used, only the stuff that needed compilation. And my fear is that this is a problem.

But how does this work without the source?

6

JIT compilation doesn’t start from source code, it starts from instantiated methods and global variables. Those objects are serialized and stored in system images and package images. Prior to v1.9 that introduced native code caching to precompiled package images, they mostly stored this.

Not 100% certain, but I think you’re right that already-compiled calls executed during the script would not provide precompile statements. However, those ought to be bundled by PackageCompiler as well. For example, any compiled native code from the default sysimage or precompiled packages are bundled; not sure if there’s another source of compiled code to worry about.