Ship custom sysimages created with PackageCompiler

[jbrea]

Currently I use PrecompilePlutoCourse to reduce latency of some Pluto notebooks for teaching. This is great, except that every student needs to wait quite a while for the PackageCompiler to finish. As there are at least an order of magnitude more students than different platforms, I thought it would be nice, if one could just distribute the custom sysimages for each platform (via Artifacts or BinDeps).

Does this sound reasonable/feasible?
Is it possible to create sysimages for other platforms on my linux machine or on some free servers in the cloud?

[SteffenPL]

It’s in development, maybe not final yet, but there is this super nice project:

GitHub - MichaelHatherly/CuratedSystemImages.jl: Julia package providing...

[jbrea]

Thanks a lot, @SteffenPL! With your link I found a github action that looks like the solution to my second question.

[Palli]

I pre-announced CuratedSystemImages.jl, then asked:

Was it too soon? It should work?

Probably, but that’s fine. It should work.

[jbrea]

Following the CuratedSystemImages approach (thanks @mike) I got it working more or less (also thanks to the great julia package manager ;)). But now I have some follow-up questions:

1. I had to patch quite a few packages that use @__DIR__ in their __init__() function with the RelocatableFolders approach. Is this the way we want to deal with relocatability in the whole ecosystem? If yes, I would open PRs for those packages.
2. Is it possible that the RelocatableFolders approach doesn’t play well with Requires? I struggled making Plots relocatable and ended up avoiding any @require, but I didn’t investigate. EDIT: all good.
3. The ubuntu and macOS build with github actions works fine. But for the windows build I always run into LLVM ERROR: out of memory, Allocation failed. Does somebody know what the problem might be?

[mike]

@jbrea

I had to patch quite a few packages that use @__DIR__ in their __init__() function with the RelocatableFolders approach. Is this the way we want to deal with relocatability in the whole ecosystem? If yes, I would open PRs for those packages.

That package, or using Artifacts, are the only approaches that I currently know of to deal with relocation correctly. RelocatableFolders has worked well in practise for what it was made to do, but just keep in mind the limitations noted in the readme, e.g. don’t use it for really large files, use artifacts instead in those cases.

@jbrea

• Is it possible that the RelocatableFolders approach doesn’t play well with Requires? I struggled making Plots relocatable and ended up avoiding any @require, but I didn’t investigate.

Haven’t come across issues before related to @require and RelocatableFolders used in conjunction. If you manage to get a reproducible case to look at we can try and resolve it. I don’t think there should be an issue between the two, but could be the case perhaps.

@jbrea

The ubuntu and macOS build with github actions works fine. But for the windows build I always run into LLVM ERROR: out of memory, Allocation failed. Does somebody know what the problem might be?

Sorry, not hit anything like that on GHA windows runners, though I’ve not tried building anything particularly “big” on GHA. Is this with Julia 1.7 or 1.8?

[jbrea]

Thanks a lot for the prompt response!

@mike

Is this with Julia 1.7 or 1.8?

1.7. My images are pretty large, using Plots, MLJ, Flux and many more packages.

@mike

If you manage to get a reproducible case to look at we can try and resolve it.

Will try.

[mike]

@jbrea

1.7. My images are pretty large, using Plots, MLJ, Flux and many more packages.

Perhaps see if you’re able to build on a local Windows machine if you’ve not done so yet? We’ve built some pretty large images on Windows without issues, but are using much larger runners that GHA provides.