# Sysimages - 2 questions - re-use, pitfalls, validity against current environment

**URL:** <https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365>\
**Category:** Tooling\
**Tags:** package, sysimage\
**Created:** [August 30, 2021, 3:52pm UTC](https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365 "2021-08-30T15:52:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ylvain](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ylvain/32/28495_2.png) [@ylvain](https://discourse.julialang.org/u/ylvain)\
**Post date:** [August 30, 2021, 3:52pm UTC](https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365/1 "2021-08-30T15:52:17Z")

</div>

A custom sysimage seems to be the current (best?, only?) solution to the latency issue. Below I only talk about [Sysimages](https://julialang.github.io/PackageCompiler.jl/dev/sysimages.html#sysimages-1), not about [Apps](https://julialang.github.io/PackageCompiler.jl/dev/apps.html#apps-1) or libraries. Contrary to the `Apps` case, I’m only concerned about weaker form of “relocability”. I envision a given sysimage being (re)-used only in very similar environment: same OS, same version of Julia, same hardware (even if _not_ same host), same depot, same artifacts, access to the same registries etc.

I see two different use cases:

1. An `app` oriented image ( **AOI** ): the build environment (as defined by `Pkg`, that is `Project`+`Manifest`) is fixed. The image will only be used with this environment. It contains an “application” package that is driven by a script. Usage is `julia -Jmyapp.so myscript args...`.
2. A `dev` oriented image ( **DOI** ): it will be used as a replacement of the default sysimage for a large range of use cases. REPL session, kernel for Juno or Jupyter, running test quickly in CI…

Sysimage have the [well known problem](https://julialang.github.io/PackageCompiler.jl/dev/sysimages.html#Drawbacks-to-custom-sysimages-1) that they break the semantics of the environment of the current active project.

However, in the first case (AOI) this is not an issue by definition. All packages in the image behave essentially like an extended standard library. From the viewpoint of the user of the image, an update of the image is like an update of the Julia distribution itself. This can be handled by the local maintainer of the image.

**Question** : is there some good tooling to help the maintainer of a set of AOI images?

- If yes, I’ve not seen it mentioned so it does not seem very popular, why?
- If no, why? is it “only” a question of manpower/priorities (of course completely understandable for a project like julia)? Or is this fundamentally misguided?

_Note 1_: I’m aware of the (for now) limited custom image feature of the VsCode extension. Also of at least one package to help create custom kernel for IJulia, and some discussions here…

_Note 2:_ I haven’t got the impression that this is “pushed” very much to the users. Maybe because of the lack of tooling? Maybe because it requires using PackageCompiler and have a “somewhat good” grasp of the Julia runtime (at least to be clear about the difference and pitfalls between the AOI and DOI use cases)

I’ve also thought a bit about the second case. I believe **DOI** image could be made to work well in practice by separating the set of packages they contain into two different parts:

1. The public interface: these packages are the ones meant to be used by the client of the image. The image should be used _only_ in environments where the public packages are _pinned_ to exactly their version in the image. Thus, the public packages must be part of the Project (and not only Manifest) of the client environment.
2. The internal interface: all the non-public packages. These will mostly be the transitive dependencies of the packages for which of the image was built (i.e. the arguments given to `create_sysimage`). We do not want all packages in the image to be part of the Project of the client, and we certainly _do not_ want them to be pinned! (this would reduce the DOI case to nearly the AOI one). The solution here is to use side-loading (which is common for runtime using so/dll). I think it could be implemented by a “clever” use of the Pkg+PackageCompiler. Of course the cost would be code duplication each time an internal package is added to the environment of the client (since the runtime would see it as a _different_ package, i.e. different uuid).

**Question** Does the above idea “kind of make sense”? As it already been discussed ad-nauseam and found lacking? How? Why?

This post is already way too long… If the idea of a sysimage builder that make the DOI use case workable is not too silly I could describe how I think one could do points 1&2 above.

(I’m _quite_ motivated, I’ve users in a enterprise scenario that would require the full DOI case with guarantee that the runtime honor their Project+Manifest specs, as it does without a custom image. Also subject to another post if anyone is interested)

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [September 1, 2021, 12:30pm UTC](https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365/2 "2021-09-01T12:30:00Z")

</div>

Re- “dev oriented images”: in our [`julia_pod`](https://github.com/beacon-biosignals/julia_pod) tool for spinning up a REPL on a kubernetes pod, we’ve adopted the convention that any pinned packages (from `Pkg.pin`) get baked into the sysimage, and any unpinned packages are totally stripped from the Manifest (i.e. if you have a pinned package that depends in some way on an unpinned package, the pinned one gets removed too). That convention means: you are free to change versions/branches/develop any unpinned package and should be fairly safe that your changes will be respected, but if you pin the bottom of the stack (that you aren’t developing/changing) then you can get the latency benefits.

A better solution (discussed on Slack and mentioned [here](https://github.com/beacon-biosignals/julia_pod/pull/9#issuecomment-897683523) would be for PackageCompiler to save a manifest of packages it has compiled, and then to teach Pkg.jl to automatically treat those packages as pinned. That way you can be sure that your manifest aligns with reality.

I think there might be an even more general solution that could handle things like `Pkg.activate(env1); Pkg.add("PackageA"); Pkg.activate(env2); Pkg.add("PackageB")` correctly would be to store the version information of any package loaded into the Julia session in the session itself and have Pkg treat it as fixed.

---

<div class="post-metadata">

**Author:** ![ylvain](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ylvain/32/28495_2.png) [@ylvain](https://discourse.julialang.org/u/ylvain)\
**Post date:** [September 3, 2021, 10:40am UTC](https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365/3 "2021-09-03T10:40:13Z")

</div>

> [@ericphanson](#):
>
> we’ve adopted the convention that any pinned packages (from `Pkg.pin` ) get baked into the sysimage, and any unpinned packages are totally stripped from the Manifest (i.e. if you have a pinned package that depends in some way on an unpinned package, the pinned one gets removed too).

Interesting simple solution. If I understood, then the creator of the compiled env. must then pin all transitive dependencies of the packages that are “really” wanted. All those dependencies will be pinned, but may not be listed in the Project.toml (since the pin status is stored in the Manifest anyway). This makes me worry about two points,

- if you include some “big” package in the sysimage (e.g. Makie), you’ll have a lot of pinned dependencies. Is the resulting env. still “extendable” in practice? Meaning, can a user, in practice, add another “big framework” package without running into unresolvable dependencies ? (I’ve not enough experience to guess)

- Client package (being dev by users) will “inherit” of this implicit env. They may start with Manifest of the build env, but what happens if they work in a “moving” git repo ? (the Manifest is in general in .gitignore, so it will regenerated and the pinning flag wil be lost, I think)

> [@ericphanson](#):
>
> A better solution (discussed on Slack and mentioned [here](https://github.com/beacon-biosignals/julia_pod/pull/9#issuecomment-897683523) would be for PackageCompiler to save a manifest of packages it has compiled, and then to teach Pkg.jl to automatically treat those packages as pinned. That way you can be sure that your manifest aligns with reality.

That’s the kind of things I wanted to do. Storing it inside the sysimage by simply generating a small “SysimageData” package included in the build, this package having all needed infos as const fields of immutable values. The SysimageData module could also have an ` __init__ ` that installs some hook to check the current env against these infos. I’ve not tested this at all for now, but it may be doable from “user-space” only (i.e. without touching Base/Pkg/PkgCompiler etc.)

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [September 3, 2021, 10:49am UTC](https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365/4 "2021-09-03T10:49:53Z")

</div>

> If I understood, then the creator of the compiled env. must then pin all transitive dependencies of the packages that are “really” wanted.

No, we don’t require this. We only remove transitive deps of _unpinned_ direct dependencies, but (unpinned) transitive deps of pinned dependencies we leave alone, unless they are also the transitive dep of an unpinned direct dep. Hopefully that makes sense aha.

> if you include some “big” package in the sysimage (e.g. Makie), you’ll have a lot of pinned dependencies. Is the resulting env. still “extendable” in practice? Meaning, can a user, in practice, add another “big framework” package without running into unresolvable dependencies ? (I’ve not enough experience to guess)

Haven’t run into issues with this yet but it’s a pretty new system.

> Client package (being dev by users) will “inherit” of this implicit env. They may start with Manifest of the build env, but what happens if they work in a “moving” git repo ? (the Manifest is in general in .gitignore, so it will regenerated and the pinning flag wil be lost, I think)

We commit Manifest.tomls for “applications” but not for “libraries”. The distinction is sometimes blurry but often as developers we are working on an “application” that uses a bunch of libraries that are baked into the sysimage for that application (and when we are working on a library, we ideally don’t really need a sysimage if it’s relatively modular and not too heavy). But agreed, to keep the pins around you need the Manifest.

> That’s the kind of things I wanted to do. Storing it inside the sysimage by simply generating a small “SysimageData” package included in the build, this package having all needed infos as const fields of immutable values. The SysimageData module could also have an **init** that installs some hook to check the current env against these infos. I’ve not tested this at all for now, but it may be doable from “user-space” only (i.e. without touching Base/Pkg/PkgCompiler etc.)

Hm, I’m not sure. I think to have a completely “foolproof” system that you can’t trick by doing stuff like activating one environment and then another one etc, one might need to modify Pkg, but probably there’s a way to get a decent working version without it.

---

<div class="post-metadata">

**Author:** ![ylvain](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ylvain/32/28495_2.png) [@ylvain](https://discourse.julialang.org/u/ylvain)\
**Post date:** [September 3, 2021, 4:44pm UTC](https://discourse.julialang.org/t/sysimages-2-questions-re-use-pitfalls-validity-against-current-environment/67365/5 "2021-09-03T16:44:38Z")

</div>

> [@ericphanson](#):
>
> No, we don’t require this. We only remove transitive

Ah ok, I had missed the bit below from your first reply

> [@ericphanson](#):
>
> (i.e. if you have a pinned package that depends in some way on an unpinned package, the pinned one gets removed too)

For my users (they are not dev) I’d need the converse strategy: they would only have to pin the package that are explicits for them (in Project, what they see by `]st`). There’s only a few. I would then pin all the transitive closure of their dependencies. So I would auto-generate a lots of pin. That’s why I worry about the “update-ability” of the resulting env. Guess I’d have to auto-rebuild the image often enough…

> [@ericphanson](#):
>
> I think to have a completely “foolproof” system that you can’t trick by doing stuff like activating one environment and then another one etc

Yeah… you cannot have anything “foolproof” in Julia since the full mutable state of Base is at your disposal 😅 … but something _intern proof_ would already be a big win for me…

I think you can auto-check the current env to emit warning/error if you override the Pkg API that update it (update, activate, add), just by some wrapper code that execute a post check.

You cannot protect against mutation of ACTIVE\_PROJECT, LOAD\_PATH, DEPOT\_PATH etc… and I feel uncomfortable intercepting Base.require (to say nothing of the s\*\*\*load of invalidations that may imply). But checking the interactive use of pkg seems doable.