Hi folks,
you might have noticed my previous announcement on Comonicon.jl, I planned to support distributable system images for CLIs so users can directly download the system images instead of building it themselves locally.
I want to share some of my explorations here, and seek some help in this direction.
What I have tried so far
Since Julia compiler do not support cross-compilation, I need to build the system image with cpu_target=x86_64 on different systems for macOS and Linux (unfortunately I have no idea how to support windows)
the first thing that came to my mind is to use GitHub Actions, thus I code up the following GitHub workflow
https://github.com/Roger-luo/IonCLI.jl/blob/master/.github/workflows/sysimg.yml
which will automatically be triggered by a push with tag, then it will
- build the system image
- pack it as tarball
- put it into the release asset
The user then can just download the system image from the release asset via a build script inside deps/build.jl automatically. However, there are some serious problems with this workflow:
since our current package registration workflow calls the JuliaRegistrator first, then the TagBot will create the tag automatically. I will not be able to register the version that contains the correct SHA of the tarball (since the tarball is generated after the tag)
On the other hand, I just found that the tag created by TagBot seems not to be a push: tag which does not trigger the GitHub Action at all.
On slack, Chris de Graaf suggests that I can think about using a custom workflow that builds the tarball first then store the tarball somewhere then trigger the JuliaRegistrator and then upload it when TagBot creates the tag.
But it’s still unclear to me how to do this. I’m wondering if anyone has an idea?
Make sure that you have configured TagBot to use an SSH deploy key.
Ah right! thanks! I didn’t notice that. Now at least TagBot triggers my github action!
I wonder if BinaryBuilder.jl can help.
BB is not a Julia compiler, so BB can’t do much on this according to staticfloat.
evalparse via JuliaLang <julialang@discoursemail.com> 于2020年8月2日周日 上午7:55写道：
I built a similar system in the past which worked pretty well for internal use, but I realized that the sysimages use absolute code paths, so unless the users you’re planning to serve have the same file system and username, I don’t believe it will work. Have you found a way around that? Hope so!
I think it depends on which package you are compiling.
This system image here is not made for general purpose, but in a very limited way - the behaviour of this system image is limited to the CLI, and the behaviour of the CLI is restricted by an eDSL compiler (Comonicon.jl).
it’s more like an app rather than a system image. I’m not building an app mainly because system image is currently much smaller (on CI, it builds without incremental build and filters unused stdlibs, the tarball is about 20MB). But I think once PackageCompiler has the ability to filter out LLVM and openBLAS, I will consider to let CLI packages build by Comonicon to generate executable binaries.
On the other hand, all the paths inside the CLI package should use Comonicon.PATH.project module, which always use a relative path to the package location.
In this way, I currently find at least the IonCLI has no installation problem using system image. But I can imagine, if someone don’t handle the path carefully, it can easily break on someone else computer, but as a result this will make things fail on CI, then you will know which part is wrong.
And if all these doesn’t work (as a result, CI fails, no tarball generated), Comonicon will give the user a clear error that the user should use mod.comonicon_build() to build the system image locally.
Hope this helps.
oh and I forget to mention that, you should never use a global constant for path, since that will not get updated on users machine.
One should always use a function with or without arguments to get the path. In my case, I always use pathof(mod) to get the current module’s path dynamically.
the implementation of PATH.project looks like this
function project(m::Module, xs...)
path = pathof(m)
path === nothing && return dirname(Pkg.project().path)
return joinpath(dirname(dirname(path)), xs...)
end
it dynamically use Pkg or pathof to get the path instead of naively use a constant, which make the paths to be correct. I think.