Sysimages - 2 questions - re-use, pitfalls, validity against current environment

Tooling
,
1

A custom sysimage seems to be the current (best?, only?) solution to the latency issue. Below I only talk about Sysimages, not about Apps or libraries. Contrary to the Apps case, I’m only concerned about weaker form of “relocability”. I envision a given sysimage being (re)-used only in very similar environment: same OS, same version of Julia, same hardware (even if not same host), same depot, same artifacts, access to the same registries etc.

I see two different use cases:

  1. An app oriented image (AOI): the build environment (as defined by Pkg, that is Project+Manifest) is fixed. The image will only be used with this environment. It contains an “application” package that is driven by a script. Usage is julia -Jmyapp.so myscript args....
  2. A dev oriented image (DOI): it will be used as a replacement of the default sysimage for a large range of use cases. REPL session, kernel for Juno or Jupyter, running test quickly in CI…

Sysimage have the well known problem that they break the semantics of the environment of the current active project.

However, in the first case (AOI) this is not an issue by definition. All packages in the image behave essentially like an extended standard library. From the viewpoint of the user of the image, an update of the image is like an update of the Julia distribution itself. This can be handled by the local maintainer of the image.

Question: is there some good tooling to help the maintainer of a set of AOI images?

  • If yes, I’ve not seen it mentioned so it does not seem very popular, why?
  • If no, why? is it “only” a question of manpower/priorities (of course completely understandable for a project like julia)? Or is this fundamentally misguided?

Note 1: I’m aware of the (for now) limited custom image feature of the VsCode extension. Also of at least one package to help create custom kernel for IJulia, and some discussions here…

Note 2: I haven’t got the impression that this is “pushed” very much to the users. Maybe because of the lack of tooling? Maybe because it requires using PackageCompiler and have a “somewhat good” grasp of the Julia runtime (at least to be clear about the difference and pitfalls between the AOI and DOI use cases)

I’ve also thought a bit about the second case. I believe DOI image could be made to work well in practice by separating the set of packages they contain into two different parts:

  1. The public interface: these packages are the ones meant to be used by the client of the image. The image should be used only in environments where the public packages are pinned to exactly their version in the image. Thus, the public packages must be part of the Project (and not only Manifest) of the client environment.
  2. The internal interface: all the non-public packages. These will mostly be the transitive dependencies of the packages for which of the image was built (i.e. the arguments given to create_sysimage). We do not want all packages in the image to be part of the Project of the client, and we certainly do not want them to be pinned! (this would reduce the DOI case to nearly the AOI one). The solution here is to use side-loading (which is common for runtime using so/dll). I think it could be implemented by a “clever” use of the Pkg+PackageCompiler. Of course the cost would be code duplication each time an internal package is added to the environment of the client (since the runtime would see it as a different package, i.e. different uuid).

Question Does the above idea “kind of make sense”? As it already been discussed ad-nauseam and found lacking? How? Why?

This post is already way too long… If the idea of a sysimage builder that make the DOI use case workable is not too silly I could describe how I think one could do points 1&2 above.

(I’m quite motivated, I’ve users in a enterprise scenario that would require the full DOI case with guarantee that the runtime honor their Project+Manifest specs, as it does without a custom image. Also subject to another post if anyone is interested)

2

Re- “dev oriented images”: in our julia_pod tool for spinning up a REPL on a kubernetes pod, we’ve adopted the convention that any pinned packages (from Pkg.pin) get baked into the sysimage, and any unpinned packages are totally stripped from the Manifest (i.e. if you have a pinned package that depends in some way on an unpinned package, the pinned one gets removed too). That convention means: you are free to change versions/branches/develop any unpinned package and should be fairly safe that your changes will be respected, but if you pin the bottom of the stack (that you aren’t developing/changing) then you can get the latency benefits.

A better solution (discussed on Slack and mentioned here would be for PackageCompiler to save a manifest of packages it has compiled, and then to teach Pkg.jl to automatically treat those packages as pinned. That way you can be sure that your manifest aligns with reality.

I think there might be an even more general solution that could handle things like Pkg.activate(env1); Pkg.add("PackageA"); Pkg.activate(env2); Pkg.add("PackageB") correctly would be to store the version information of any package loaded into the Julia session in the session itself and have Pkg treat it as fixed.

1 Like
3

Interesting simple solution. If I understood, then the creator of the compiled env. must then pin all transitive dependencies of the packages that are “really” wanted. All those dependencies will be pinned, but may not be listed in the Project.toml (since the pin status is stored in the Manifest anyway). This makes me worry about two points,

  • if you include some “big” package in the sysimage (e.g. Makie), you’ll have a lot of pinned dependencies. Is the resulting env. still “extendable” in practice? Meaning, can a user, in practice, add another “big framework” package without running into unresolvable dependencies ? (I’ve not enough experience to guess)

  • Client package (being dev by users) will “inherit” of this implicit env. They may start with Manifest of the build env, but what happens if they work in a “moving” git repo ? (the Manifest is in general in .gitignore, so it will regenerated and the pinning flag wil be lost, I think)

That’s the kind of things I wanted to do. Storing it inside the sysimage by simply generating a small “SysimageData” package included in the build, this package having all needed infos as const fields of immutable values. The SysimageData module could also have an __init__ that installs some hook to check the current env against these infos. I’ve not tested this at all for now, but it may be doable from “user-space” only (i.e. without touching Base/Pkg/PkgCompiler etc.)

4

If I understood, then the creator of the compiled env. must then pin all transitive dependencies of the packages that are “really” wanted.

No, we don’t require this. We only remove transitive deps of unpinned direct dependencies, but (unpinned) transitive deps of pinned dependencies we leave alone, unless they are also the transitive dep of an unpinned direct dep. Hopefully that makes sense aha.

if you include some “big” package in the sysimage (e.g. Makie), you’ll have a lot of pinned dependencies. Is the resulting env. still “extendable” in practice? Meaning, can a user, in practice, add another “big framework” package without running into unresolvable dependencies ? (I’ve not enough experience to guess)

Haven’t run into issues with this yet but it’s a pretty new system.

Client package (being dev by users) will “inherit” of this implicit env. They may start with Manifest of the build env, but what happens if they work in a “moving” git repo ? (the Manifest is in general in .gitignore, so it will regenerated and the pinning flag wil be lost, I think)

We commit Manifest.tomls for “applications” but not for “libraries”. The distinction is sometimes blurry but often as developers we are working on an “application” that uses a bunch of libraries that are baked into the sysimage for that application (and when we are working on a library, we ideally don’t really need a sysimage if it’s relatively modular and not too heavy). But agreed, to keep the pins around you need the Manifest.

That’s the kind of things I wanted to do. Storing it inside the sysimage by simply generating a small “SysimageData” package included in the build, this package having all needed infos as const fields of immutable values. The SysimageData module could also have an init that installs some hook to check the current env against these infos. I’ve not tested this at all for now, but it may be doable from “user-space” only (i.e. without touching Base/Pkg/PkgCompiler etc.)

Hm, I’m not sure. I think to have a completely “foolproof” system that you can’t trick by doing stuff like activating one environment and then another one etc, one might need to modify Pkg, but probably there’s a way to get a decent working version without it.

5

Ah ok, I had missed the bit below from your first reply

For my users (they are not dev) I’d need the converse strategy: they would only have to pin the package that are explicits for them (in Project, what they see by ]st). There’s only a few. I would then pin all the transitive closure of their dependencies. So I would auto-generate a lots of pin. That’s why I worry about the “update-ability” of the resulting env. Guess I’d have to auto-rebuild the image often enough…

Yeah… you cannot have anything “foolproof” in Julia since the full mutable state of Base is at your disposal :sweat_smile: … but something intern proof would already be a big win for me…

I think you can auto-check the current env to emit warning/error if you override the Pkg API that update it (update, activate, add), just by some wrapper code that execute a post check.

You cannot protect against mutation of ACTIVE_PROJECT, LOAD_PATH, DEPOT_PATH etc… and I feel uncomfortable intercepting Base.require (to say nothing of the s***load of invalidations that may imply). But checking the interactive use of pkg seems doable.

2 Likes