No, it does not work. It only uncovers a severe issue in Julia 1.11. I write in Julia because using the same PackageCompiler version works well with the latest Julia 1.10.9 (no matter if I have the environment variable “JULIA_NUM_THREADS” or not), and does not work with Julia 1.11.4.
Your patch is only a hotfix to make it compile at all, but the compilation takes up to 40% longer than with 1.10 and, more important, I often get errors that some threads cannot be stopped and the compilation fails. These issues also occur with your patch.
Something is is different in Julia 1.11 or PackageCompiler needs an update for Julia 1.11. I mean that it suddenly fails if users have set JULIA_NUM_THREADS and even with only one thread it does no longer run stable and takes way more time. With way more time, I directly compared Julia 1.11.4 and 1.10.9 using each only one thread.
I think that is @kristoffer.carlsson definition of “works”. So just to double check: independently of how long it takes to compile, it does compile and the resulting file does what you need it to do. Is that correct?
The wording isn’t the clearest and details aren’t given, but I’m pretty confident uwestoehr is saying single-threaded PackageCompiler takes longer (~40%) and often fails with errors on v1.11.4 compared to v1.10.9 for something. To broadly speculate, some third-party package developers have reported slower compilation, larger caches, or slower loading, which could be related to any of the big internal changes in v1.11 which PackageCompiler can’t directly do anything about; the silver lining is precompilation at least only has to be paid once per unique environment and there are people invested in improvements.
Exactly. During the work on my solution, I announced in this thread.
I performed dozens of compilations. With Julia 1.11.4 there are threading issues. You get during PackageCompilation first warnings that a thread could not be ended and in contrast to Julia 1.10 the compilation sometimes fails. And as strange as it is, this is the case no matter if I set JULIA_NUM_THREADS to 1 or don’t have a JULIA_NUM_THREADS at all. Also, you can perform the same compilation 10 times in a row and several times the compilation fails but not always and not reproducibly. I really have no clue and this kind of randomness made me crazy.
To test, I documented what I did in detail. I cannot do more as I have no clue how the JIT compiler works and how threading is done. After working on this many hours, I am however, sure that the issue is somewhere in the handling of threading.
Note that Julia 1.10 also gives you kind of random threading warnings, but first after the nonincremental system image was successfully compiled and they can be ignored as the compilation eventually succeeds every time.
I think that should serve as a MWE. The only things left for reporting an issue is sharing the exact text of your versioninfo(), the 1.10 threading warnings, the 1.11 errors, and relevant descriptions (do they occur sporadically, randomly, condtionally, etc). The rest should be investigated by people who know PackageCompiler or Julia internals.
Yes, we already established this, but as I said, I don’t think this has anything to do with threading. Maybe you didn’t see this because my message got moved (I think):
When generating an incremental system image on Linux, threading makes the process much faster, like 6 min instead of 10min (well, if you have enough RAM, otherwise it crashes. For me, 32GB are sufficient). On Windows multi-threading was never used because the linker does not manage to link the different files created by the different threads.
I created a PR to limit the single-threading to the one call where it is currently necessary.
I also created a Julia issue for fixing the last place where single-threading is needed. The latter one needs some in-depth mutex knowledge, I assume. But we have a commit which probably introduced the problem and we have a test case, so it sounds possible to hunt that bug down.
Please have a look if you know something about these things.
@uwestoehr: Can you please test whether the compilation takes still longer with the fix from the PR?
You can apply the fix (temporarily) manually in the packages directory of your Julia installation (sorry, I don’t know where that is located on Windows (just to prove your point from above ;-))).
Alternatively, you can point Julia’s package manager to add https://github.com/PatrickHaecker/PackageCompiler.jl#fix_hang_multiple_threads.
result: compilation works, total time 593 seconds, here a screen shot with all details. Note that there are now no warnings about threads.
But the compilation time increased by 63 %
No. The difference is only 5 seconds, so for me 368 s instead of 363 s. And you are right, the multithreading is actually only used when the packages are precompiled.
BUT, Julia 1.11.5 compiles the same module via PackageCompiler 63 % slower than Julia 1.10.9, see my detailed test here.
This means the PR should probably be integrated. @kristoffer.carlsson I Think you have the permissions to do so. Could you pleas have a look?
The long build times are also annoying for us (in the company I work for). But as the development where fast iterations are necessary works without PackageCompiler.jl they are only that: annoying, but not really expensive.
Therefore, we don’t have a business incentive to invest into making it faster. My impression from looking at the code and its history is that this is the same for others, too. @kristoffer.carlsson has done a tremendous job with creating and maintaining PackageCompiler.jl together with the community, but it does look like it was never really a focus topic for a longer time (which is only natural with so much on the plate).
The run-time of PackageCompiler.jl could be reduced by more than a factor of two by caching some of the artefacts and I am pretty sure there is even more optimization potential when going down the rabbit hole. However, I guess people will mostly focus on the --trim feature in 1.12 and beyond.
Does your company, @uwestoehr, have a relevant business incentive for fast compilations and the possibility to pay someone improving PackageCompiler.jl or --trim? I assume more hands would be more than welcome. Otherwise I think we need to be thankful for all the great work people in the community do and try to help out whenever we have some time to spare.
At first I have to convince the team that my approach to use neuronal nets can improve the overall performance of the sensors. And then to prove that my idea to use Julia’s FluxML, compile the Julia code to a DLL and then use it for a UI actually works.
As of now I don’t know if my idea will work.
