Julia image threads on Windows

General Usage
,
1

As far as I know, by default PackageCompiler uses JULIA_IMAGE_THREADS=1 on Windows and all cores on Linux (Julia v1.10) due to which it completes much faster on Linux, but uses more RAM. I set on Windows system variable to 12 which is my number of cores, but that did not appear to do anything. Does Julia support using more than 1 image thread on Windows? Is this missing feature or a bug? Can Linux behavior of parallelism be recreated on Windows?

2

It does not support that because of limitations of the linker on Windows. The parts created by different threads need to be linked together. I do not know if a work-around for this problem would be possible. See: High number of external symbols in Windows 10 when creating sysimages with julia 1.10.0-beta1 ¡ Issue #50954 ¡ JuliaLang/julia ¡ GitHub

The open - and unanswered - question in this issue is, why does the package compiler add --export-all-symbols to the linker parameters which results in too many symbols.

2 Likes
3

Thank you for detailed answer. I am looking forward if someone will be fixing this.

Besides functionality, it is not in documentation that this is not working on Windows.

4

Perhaps you can create an issue “Missing support for multi-threaded system image creation on Windows” in the package PackageCompiler.jl ? To make this work probably changes to both PackageCompiler.jl and Julia are required.

5

People are working on fixing this, and if I’m reading correctly, have already done so on master that you can try checking out, I’m not on Windows to test, but curious if it works):

using Pkg

Pkg.add(PackageSpec(url="https://github.com/JuliaLang/PackageCompiler.jl", rev="master"))

in case I'm wrong about master, check out that PR/commit:

Pkg.add(PackageSpec(url="https://github.com/JuliaLang/PackageCompiler.jl", rev="0f2d3b70cff4a72fd77d0af68de93e2491abd346"))

Note, it was limited to 1 thread, by design, as a workaround, that’s now apparently no longer needed:

This was a problem on 1.11, but less of a problem on 1.10? I hope both fixed now, and if not try either with fewer threads > 1.

Julia does for sure support threads on Windows too. This was only a problem while compiling. I think, though not sure, that you could compile with 1 thread to make the compiling work, but then when using that compiled code have as many threads as you like.

6

No, it does NOT support multi-threaded system image generation on Windows. The issue you are referring here is a very different issue. It was more an issue on Linux when creating a non-incremental system image. Creating an incremental system image using multiple threads on Linux always worked and still works, at least with Julia 1.10 and 1.11. It does not work multithreaded on Windows. Not sure if it was always like that or if there are old Julia versions where it still works.

7

I created the issue on PackageCompiler.jl.

1 Like