PackageCompiler compilation time > 2h

General Usage
1

I’m trying to compile my project to a standalone executable with PackageCompiler.jl.

I have used PackageCompiler before just for toying around, and I remember compilation times were slow, but now with a real codebase (20 deps, ~1k lines of code) this is taking already more than 2 hours (and counting).

Is having several hours of compilation time for PackageCompiler a normal thing, or there might be a problem in my setup?

2

Do you have enough free RAM? Which OS? On Windows virus scanners might slow you down. So far I never saw compilation times of more than 15 min on a not too old PC with projects of a few thousand LOC…

2 Likes
3

What Julia version are you using?

4

5 hours and counting… I guess I should just stop it and try to remove some dependencies o compile a simpler main function…

8GB on Linux

1.9

5

I believe version 1.10 has some new features that would speed this up. One feature is using pkgimages to help build a system image.

1 Like
6

Well, compiling the system image took 4:30 minutes.

The project also precompiled successfully, in about 20 minutes for a total of 226 dependencies (altough I have ~20 in my Project.toml).

It then got stuch in the very last phase, it seems.

I’m trying again now with a clean environment and removing some unused deps like Revise…

1 Like
7

Code block only for PackageCompiler and not for Revise - #5 by tim.holy I also learned of we should not add Revise to every package after using PackageCompiler.

1 Like
8

I tried again without Revise and stopped the compilation at 1 hour without success.

Btw, I’m also using create_app. I can compile “hello world” apps in about 5 minutes.

I also tried to compile a test application that only includes Plots:

module test_packagecompiler
using Plots
function julia_main()::Cint
    println("Generating random plot and saving to file...")
    heatmap(rand(100,100))
    savefig("random_heatmap.png")
    return 0
end
end

… and after 50 minutes, I gave up.

However, I removed Plots from one of the packages I was trying to compile, and it compiled in about 12 minutes!

A funny thing happened, though. The first time I ran it, it took some time in downloading some artifact (MKL) which had apparently forgot to include during the build, but the second time I ran it, it went straight into the results.

So I guess no plotting on compiled apps? That’s curious, because the first thing I ever did with PackageCompier was to add Plots to a sysimage… Why would something work for a compiled sysimage and not for a standalone app?

1 Like
9

Is it possible that one of your precompilation scripts was waiting for input?

10

8Gbytes is not a lot of RAM these days but should be enough.
I would advise openign a new terminal and running these utilities as the compilation is happening
top
htop
iotop

Often a simple ‘top’ will give you insight into what is going on.

1 Like
11

Also please look at this thread

This is twice we have a report of very large compilation times. Are we seeing the start of a common problem?

I suspect this is not a bug in Julia, rather that these systems are swapping due to large memory use. Thats just a wet finger stuck up to test the wind of course.
Please report versioninfo()

12

There is an open issue for PackageCompiler.jl PackageCompiler spins infinitely when compiling sysimage · Issue #825 · JuliaLang/PackageCompiler.jl · GitHub So far, it seems that having Plot.jl or StatsPlots.jl included in the image leads to problems, but why this is so is not clear. With Julia 1.8.x, the same projects do compile successfully, in a “normal” amount of time.

2 Likes
13

Just don’t try to compile Plots with PackageCompiler. It is very difficult to make it work and probably not worth the effort. There are lots of other plotting packages that cause less problems.

14

Yes, it seems that this is a regression somewhere, maybe in Plots or PackageCompiler… because it definitively used to work.

Just for the record:

Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × AMD Ryzen 5 1400 Quad-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, znver1)
  Threads: 1 on 8 virtual cores
15

See the issue I mentioned above, there is a work around for julia 1.10rc2, which I suppose will probably not be needed for much longer.

1 Like
16

Yes, I’m watching that issue, thanks for the reference. I couldn’t try it yet, but I’ll report back when I do.