If I’m not mistaken if you use a custom sysimage and then make any changes to any of the packages included in said sysimage, then these won’t be reflected in any Julia sessions where you use that sysimage. What I’m wondering is whether if there is any way to detect such ‘invalidations’ of the sysimage. So kind of thinking of the sysimage of something of a cache.
If not currently possible is there a way to get there? I’m thinking something like including some kind of hash of the package sources that went into creation of the sysimage.
And kind of a related question, is the sysimage necessarily monolithic? Or can it be decomposed into some number of parts, such as by package, such that ideally if one package in an environment that was used to create the sysimage was updated, just that part of the sysimage could be recreated?
[…] note that everything [i.e. the process of detecting the correct sysimage to use, note by me] has to be implemented in C (or C++ or scheme) because we can’t use Julia before loading the system image [^1]. So, that’s why I’m shooting for a very simple and flexible interface in this PR.
[^1] OK, technically, we can do this in a subprocess or maybe even re-initializing Julia runtime is possible? But it sounds very tricky and fragile to me.
More directly related to your question, I’ve tested something like what you mention in order to build & invalidate system images in eglot-jl. In order to have this in Julia, I used the workaround mentioned by @tkf in the above mentioned comment: everything related to sysimage detection is performed in a preliminary Julia process, which then starts a new Julia process with the correct sysimage. The relevant PR is not (yet?) merged, but I’ve used it on a daily basis for the last few months without trouble, which I think validates the practical usefulness of the approach.
However the whole system is indeed a bit fragile; it works well in this case, but only because in eglot-jl we completely control the environment we want to work in, and the command-line flags provided to Julia. I wonder if such a technique could be robustly implemented in a more general context, but unfortunately I’ve never really taken the time to think it through…
In any case, I’m looking forward to reading what others have to say about this!
Thanks, those are some useful pointers. It sounds like there’s some work ongoing that in the longer term can really improve sysimage workflows.
In the meantime I think what I might try is a simple experiment in my own package with at script that saves a snapshot of Manifest.toml as it was when I ran PackageCompiler last and if there has been a change it will re-run PackageCompiler.
It’s actually so simple that I suspect others are already doing this but I might be missing some nuance… OTOH I also wonder whether any and all changes to Manifest.toml should trigger a rebuild or only packages that I list in the call to create_sysimage.
But they always use all packages in the environment to build the sysimage. I think the right set to watch is the packages you pass to create the sysimage, and all of their dependencies (and their dependencies, etc).
Thanks. Mostly though I don’t run from within VS Code–and it looks like that is only for the REPL in VS Code?
At any rate, just so I actually have some transparency to what’s going on–and also, IIUC, not updating the sysimage will just silently cause much confusion down the road–for now I’m going to try rolling my own script.
Oh yeah, I didn’t mean to suggest you should re-use their code (which is in typescript anyway), I just wanted to provide some support to the idea that others are already doing it for various uses . The VSCode extension does it so you can ask it to compile a sysimage through the UI and it knows when to invalidate it, and automatically start the REPL with that sysimage.
I think it makes sense to roll your own script for now, though hopefully at some point there is more tooling built up around it. In fact, hopefully at some point the package manager itself can be aware of when sysimages are invalidated by changes to the project (https://github.com/JuliaLang/Pkg.jl/issues/2008).
Yes I’m particularly excited for that to come about!
So a related thing I’m wondering: Is there a way to determine what packages (modules?) are worth including in the create_sysimage call? I mean I guess I can just list every single package in the manifest but over time I’ll start to accumulate packages that I don’t always use.
I’m also still not quite clear on under what conditions I should definitely update the sysimage. For example, do I need to worry about transitive dependencies or do these show up in the manifest as changes anyway? Are there any changes that would not show up in Manifest.toml yet would invalidate a custom sysimage in some way?
Is there a way to determine what packages (modules?) are worth including in the create_sysimage call?
I think it’s a bit of a balancing act, because you are locking in those versions, so updating them means waiting for a sysimage rebuild. So I think it’s good for packages you use often but are fairly stable and you don’t often need to update them (or their (transitive) dependencies!). I used to use a sysimage for plotting with Makie.jl, but it’s still in a state of active development and I end up updating a lot, and so I stopped using one. At my work, we also use sysimages in docker images, to package up a piece of code and have it start quickly. There we just put everything except for dev’d dependencies into the sysimage (i.e. everything except for the code we’re actually working on).
do I need to worry about transitive dependencies or do these show up in the manifest as changes anyway?
Yeah, changes to transitive dependencies will show up in the Manifest.
Are there any changes that would not show up in Manifest.toml yet would invalidate a custom sysimage in some way?
Hm, changing the Julia version would do that, but I can’t really think of anything else.
. The VSCode extension does it so you can ask it to compile a sysimage through the UI and it knows when to invalidate it, and automatically start the REPL with that sysimage.