# Automatically build a system image on package installation

**URL:** <https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907>\
**Category:** General Usage\
**Tags:** package, build, installation\
**Created:** [September 1, 2020, 3:59pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907 "2020-09-01T15:59:29Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [September 1, 2020, 3:59pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/1 "2020-09-01T15:59:29Z")

</div>

For colleagues used to a workflow centered around running Python scripts from the command line, I was wondering if a similar workflow with Julia could be facilitated by automatically building a system image containing relevant packages when the Julia package is installed. Has anyone tried this before? Is it a bad idea?

I was reading in the docs for the pyjulia interface that it supports using a Julia system image, but provides instructions only for how to manually build one.  
[https://pyjulia.readthedocs.io/en/latest/sysimage.html](https://pyjulia.readthedocs.io/en/latest/sysimage.html)

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [September 1, 2020, 4:53pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/2 "2020-09-01T16:53:48Z")

</div>

I regularly compile a bunch of packages. As long as it doesn’t compile for every new pacakge but does it bachted then performance won’t be that bad.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [September 1, 2020, 5:08pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/3 "2020-09-01T17:08:41Z")

</div>

I was more thinking of providing a Julia package which my colleagues could use that would automatically build the needed system image so that it’s snappy to use when they call it from Python, possibly from the command line. The system image build would thus be done when this particular package was installed

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 1, 2020, 6:35pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/4 "2020-09-01T18:35:41Z")

</div>

A few topics that seem related:

[https://github.com/JuliaLang/julia/pull/35794](https://github.com/JuliaLang/julia/pull/35794)

[https://github.com/JuliaLang/julia/issues/32906](https://github.com/JuliaLang/julia/issues/32906)

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 1, 2020, 7:02pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/5 "2020-09-01T19:02:17Z")

</div>

I think that’s an interesting idea. But if you want to provide something that works fully automatically (for example by replacing Julia’s default sysimage), you’ll probably have to think carefully about what you want to happen when one of the packages baked into the sysimage gets updated. The situation you’d want to avoid is one where your colleagues would update the packages they use but, because they keep using an out-of-date sysimage, would not actually get the behavior of updated packages.

One thing I think could work, would be to install in their `PATH` a small script called `julia` and acting as a proxy to the real julia binary. The script would check the julia version, the versions of all relevant packages, and see whether it finds a suitable sysimage. If yes, it runs julia with the sysimage. If not, it warns the user and/or proposes to create a sysimage.

We’re currently trying to use a similar mechanism to provide sysimages for `LanguageServer` to be used by [`eglot-jl`](https://github.com/non-Jedi/eglot-jl). The way we are currently experimenting for doing this is by using a naming convention for the sysimage, that includes the julia version as well as a hash of the Manifest contents. If you want to look at the details, the relevant PR is here:

[https://github.com/non-Jedi/eglot-jl/pull/12](https://github.com/non-Jedi/eglot-jl/pull/12)

VScode people probably have much more experience about this. They seem to tackle the same problem by [comparing the last modified dates of the sysimage and the Manifest](https://github.com/JuliaLang/julia/pull/35794#issuecomment-652092785).

And I guess I should also say that all of these techniques can fail in some way or another if you don’t have enough control on the environment that users want to use (stacked environments might cause issues, for example).

  

While I may be implying that the process is not entirely straightforward, I wouldn’t want to deter you from trying to implement this. And I’d very much like to hear about your progress!

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [September 2, 2020, 1:57pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/6 "2020-09-02T13:57:33Z")

</div>

I just want to mention that there are users that would never themselves run julia. I often write tools for people with zero programming knowledge. These people do not know or need to know how to even start the REPL. They just want a “program”. It would be amazing to have the option to wrap an application (i.e. a package with a Manifest file that basically exports just one `main` function) with some machinery that locally produces an executable that was compiled for and on that local system. Boom.

So yea, we would need some way for them to install that package, and some conventions on where to place that executable, but it would mean that if I made a tool (that passed enough testing, vetting, etc, i.e. I don’t want to update it every week) I’d be able to get the users to “install” (which means internally, Pkg.add, build, BB the app, with the usage file, and link the binary to some shortcut icon on their desktop or some such) it and use it with maximum speeds.

But as I said, it would be mostly useful for people that don’t need to start the REPL.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [September 2, 2020, 2:14pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/7 "2020-09-02T14:14:41Z")

</div>

> [@ffevotte](#):
>
> you’ll probably have to think carefully about what you want to happen when one of the packages baked into the sysimage gets updated.

I agree, but realistically, only the package I am providing would ever be updated by the user, thus triggering the recompilation.

> [@ffevotte](#):
>
> And I guess I should also say that all of these techniques can fail in some way or another if you don’t have enough control on the environment that users want to use (stacked environments might cause issues, for example).

The user would in this case not be concerned with julia environments, but rather load and use the system image from python. As soon as the user actually wants to use julia and not only call some function from python, I think a manual process is worth the effort.

> [@ffevotte](#):
>
> While I may be implying that the process is not entirely straightforward, I wouldn’t want to deter you from trying to implement this. And I’d very much like to hear about your progress!

I’ll be sure to report back if I manage to get something usable out of this 🙂 Thanks for all consideration!

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 2, 2020, 2:18pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/8 "2020-09-02T14:18:30Z")

</div>

> [@baggepinnen](#):
>
> I agree, but realistically, only the package I am providing would ever be updated by the user, thus triggering the recompilation.

Cool! Then I think using system images is a very good idea. You’re planning to trigger the creation of the sysimage in the build step of your package’s installation?

> [@baggepinnen](#):
>
> I’ll be sure to report back if I manage to get something usable out of this 🙂

Thanks!

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 2, 2020, 2:35pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/9 "2020-09-02T14:35:23Z")

</div>

> [@yakir12](#):
>
> It would be amazing to have the option to wrap an application (i.e. a package with a Manifest file that basically exports just one `main` function) with some machinery that locally produces an executable that was compiled for and on that local system. Boom.

We already have that option, no? Couldn’t you ship you application with a Julia script that:

- instantiates the environment,
- calls PackageCompiler to create a sysimage or an app,
- optionally install a desktop shortcut icon linking to the app (or a script running it)?

Provided that Julia is correctly installed on the users’ systems, installing the app should be as simple for them as downloading an archive, uncompressing it and double-clicking on the `install` script.

On your side, writing a script as described above should not be almost no work.

FWIW I’ve already used this kind of strategy a couple times, but for users who seem to be more tech-savvy than yours (and on Linux). I would ship a Makefile alongside the sources of the app, and they would clone the git repo and run `make` or `make install` to build a sysimage and install in their PATH a script running the app.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [September 2, 2020, 2:41pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/10 "2020-09-02T14:41:07Z")

</div>

If you are using Julia via pyjulia, you’d need to create sysimage after pyjulia is installed (as it has to pull off some tedious magics). Unfortunately, I don’t think Python’s package installer (`pip`) has an appropriate post-install hook mechanism to make it work. If your colleagues use conda, it’s probably possible to do something via `post-link` hook: [Adding pre-link, post-link, and pre-unlink scripts — conda-build 3.22.0+10.g432a9e1b.dirty documentation](https://docs.conda.io/projects/conda-build/en/latest/resources/link-scripts.html). Having said that, I think it’s better to do this in a Python script/package to compile the sysimage on the first execution time.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [September 2, 2020, 2:43pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/11 "2020-09-02T14:43:13Z")

</div>

Good points. I’ll most def look into this (remember I first need to have one of my tools reach this magical realm of doesn’t-need-to-be-updated-every-week).

---

<div class="post-metadata">

**Author:** ![kevbonham](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kevbonham/32/216165_2.png) [@kevbonham](https://discourse.julialang.org/u/kevbonham)\
**Post date:** [September 2, 2020, 4:04pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/12 "2020-09-02T16:04:42Z")

</div>

> [@ffevotte](#):
>
> We already have that option, no? Couldn’t you ship you application with a Julia script that:
> 
> - instantiates the environment,
> - calls PackageCompiler to create a sysimage or an app,
> - optionally install a desktop shortcut icon linking to the app (or a script running it)?
> 
> Provided that Julia is correctly installed on the users’ systems

New PackageCompiler.jl has an option to make an “app” where the user doesn’t even need Julia installed. On mobile, so can’t easily find the link, but check out Kristoffer’s JuliaCon presentation from this year.

One thing I’m not clear on is whether the app can take arguments. But regardless, it looks pretty neat.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 2, 2020, 8:39pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/13 "2020-09-02T20:39:37Z")

</div>

> [@kevbonham](#):
>
> One thing I’m not clear on is whether the app can take arguments.

It can, they get passed to the julia main function.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 3, 2020, 9:13am UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/14 "2020-09-03T09:13:07Z")

</div>

> [@yakir12](#):
>
> there are users that would never themselves run julia. I often write tools for people with zero programming knowledge. These people do not know or need to know how to even start the REPL. They just want a “program”.

While I don’t dispute the use case for standalone, self-contained, mostly precompiled “executables”, I think that in 2020 most of the people you are talking about above really want a web service, maybe on the local intranet.

---

<div class="post-metadata">

**Author:** ![yakir12](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yakir12/32/297_2.png) [@yakir12](https://discourse.julialang.org/u/yakir12)\
**Post date:** [September 3, 2020, 9:47am UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/15 "2020-09-03T09:47:00Z")

</div>

Agreed!

Having these apps work from a browser is better in many respects, but creating executable apps is often simpler, depending on the task. As mentioned above, I’d argue that it’s pretty straight forward to create an executable on the user’s local machine. Maybe soon, what with JSServe etc, it might be equally easy to build a webapp.

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 3, 2020, 11:35am UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/16 "2020-09-03T11:35:33Z")

</div>

> [@Tamas\_Papp](#):
>
> I think that in 2020 most of the people you are talking about above really want a web service

I tend to agree on that point, especially when all user inputs can be nicely presented in a web UI and the output can be nicely presented in the UI, or downloaded as a data file to be further analyzed by another program.

However I’m under the impression that there are things that are difficult to do with web services. I’m thinking in particular about I/O from/to multiple files on the user’s computer, as would be needed by a more complex program that looks for data/parameters in multiple input files, or produces several different and complex outputs.

(But I don’t really know anything about such topics, so I might be wrong. And I guess a lot of use cases are covered by a simple process where the user uploads one input file, the “program” runs remotely, and the user subsequently downloads a results file)

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [December 15, 2021, 7:22pm UTC](https://discourse.julialang.org/t/automatically-build-a-system-image-on-package-installation/45907/17 "2021-12-15T19:22:28Z")

</div>

This package does something like what you are asking for

[https://github.com/jlapeyre/julia\_project](https://github.com/jlapeyre/julia_project)

I already posted about it. But, I think it’s worth posting here because it addresses the issue.

julia\_project is a Python package. You use it inside another Python package and module to manage the Julia dependency. There is a method `compile_julia_project()` that you call to make a system image that will be found the next time you import the Python module. My operating assumption is that people love Python and feel comfortable with it, so asking them to call a Python method is a very small ask. On the other hand Julia is unknown to them, and they don’t have time to be exposed to anything new. However, if your users don’t even like to be told to call a Python method , you might automate it. I did not add a keyword option to automatically compile on installation. But, that would be an easy modification.  
It works like this in your Python module code:

```python
import os
mymodule_path = os.path.dirname(os.path.dirname(os.path.abspath( __file__ )))

julia_project = JuliaProject(
    name="mymodule",
    package_path=mymodule_path,
    registry_url = "git@github.com:myuser/MyModuleRegistry.git",
    logging_level = logging.INFO # or WARN, or ERROR
    )

julia_project.run() # This exectutes all the management features.

def compile_mymodule():
    julia_project.compile_julia_project()

```