How should such a script be packaged? The obvious answer I can think of is as a regular Julia package, with a bin subdirectory containing the necessary scripts. This pattern is already used by some packages, but there are certain conventions that should be established, e.g.:
should they keep the .jl extension, use a different extension (.jlx?), or use no extension?
How should a user manage such packages? Ideally, installation should be a single step, and result with the script name appearing in the users PATH, and should have a simple mechanism for upgrading or removing it. There have been a couple of different approaches either used or suggested:
MPI.jl provides a function which copies bash script into ~/.julia/bin: this both requires manual intervention to install or update.
Doing some magic at Pkg.build time: I think this should be discouraged, see here for some earlier discussion of this issue: Project with scripts (bin/)
I think the cleanest solution would be to have it tied to a specific environment, i.e. installing a package in this environment would result in the script be accessible via the PATH. How to make this work–especially across multiple platforms–is a difficult question.
How could this system integrate with (a) PackageCompiler.jl (e.g. to compile a specific script or system image), and (b) system package managers (e.g. so that a script could be installed via apt, yum or homebrew)?
Any thoughts or ideas on how we might improve things?
Unix scripts may have shebang #!/usr/bin/env julia.
CLI tools using a system’s existing julia installation
Declaratively define the set of installed applications in ~/.julia/config/applications.toml.
# Define a project for linting that has access to StaticLint
# and an optional dependency of it.
[applications.linting]
juliapath = "/usr/local/julia-1.7.1/bin/julia"
[applications.linting.deps]
StaticLint = "b3cc710f-9c33-5bdb-a03d-a94903873e97"
AnOptionalStaticLintHelper = "571b7581-3216-41df-b45e-703c535b21e8"
# Avoid conflicting binary names by specifying links.
[applications.linting.link.mapping]
"/StaticLint/bin/staticlint" = "jlint-classic"
"/AnOptionalStaticLintHelper/bin/staticlint" = "jlint-helper"
Installation creates an applications_manifest.toml.
% julia
Pkg> app install --edit-profile
Installed:
~/.julia/bin/jlint-classic -> ~/.julia/packages/StaticLint/ABCDE/bin/staticlint
~/.julia/bin/jlint-helper -> ~/.julia/packages/AnOptionalStaticLintHelper/FGHIJ/bin/staticlint
The directory ~/.julia/bin was added to your shell profile.
The next time you log in the programs should be available.
^D
% ls -l ~/.julia/bin/
~/.julia/bin/jlint-classic -> ~/.julia/packages/StaticLint/ABCDE/bin/staticlint
~/.julia/bin/jlint-helper -> ~/.julia/packages/AnOptionalStaticLintHelper/FGHIJ/bin/staticlint
# Next session, the PATH is changed.
% echo $PATH
~/.julia/bin/:~/.local/bin/:...
Using a system’s Julia compiler can cause issues that static binaries don’t have. Unlike ~/.julia/packages, the set of installed julias isn’t controlled by the Pkg team. How small can Julia’s standalone sans-llvm binaries be?
I think although it’s not perfect but under current package system we can already do something. Here’s what Comonicon does:
an install function is provided to install the CLI scripts
all CLI scripts live in ~/.julia/bin is tied to a specific environment created in corresponding package’s scratch space (using Scratch.jl), this is because:
in order to make compile cache work one will need to define most of the implementation in a project module (maybe in the future we can also do some compile cache for specific type of script file instead of project modules only?) this is quite crucial for a non-trivial CLI to work with low latency (yes after many times of trying I find supporting CLI scripts directly with custom compile cache doesn’t help much for complicated scripts, the best way is still to write project modules, so this feature is removed from Comonicon in recent versions)
to make system image work the CLI needs to be tied with a specific Manifest.toml
the scratch space can be properly removed after the package is removed
different CLI shouldn’t share one environment because they may depend on different versions and they shouldn’t use the package environment because we want to fix the dependencies and use other package to generate related method cache (e.g to support CLI plugin) (a mutable Manifest.toml file instead of immutable Project.toml) once it’s installed.
A config file Comonicon.toml is used to define the default behavior of installation and building, e.g compile options for starting the CLI and for compiling using PackageCompiler (nthreads, compile=min, etc)
The CLI package author can choose to run installation by default via putting the install function in deps/build.jl (This is not ideal tho)
The installed scripts however is not a julia script but a small shell script + a small julia script, this is because to make the latency reasonable one won’t want to have a lot of code in a single Julia script. And the bash script is used to config the start up option (the environment path, system image path etc)
The shell autocomplete are generated if the default shell is supported (currently only ZSH) then the completion scripts are installed at ~/julia/completions
If the author choose to install with system image (either via local build or download prebuilt system image binary from GitHub release CDN), the system image will be installed into the scratch space of corresponding package too. This makes sure when one updates or remove the package system image can get updated.
Package author can also choose to build applications which compiles everything into a binary with a bunch of shared libraries (via PackageCompiler.create_app) and pack them into a tarball. This workflow is also supported with install function (itself actually is also a CLI). So after compile the whole thing then upload it to GitHub release CDN nothing is tied to Pkg or julia anymore it’s completely standalone (so can be used as an artifact in principle)
So a few things I’m not happy with the current mechanism Comonicon uses as a workaround due to limitations of Pkg
since there’s no mechanism for installing things that has a build process of system image, users will not see a progress bar of building or downloading system image if the author choose to use a system image for the CLI (because all the info from build.jl will be redirected, unless with verbose option)
the installed scripts in .julia/bin and .julia/completions will not be deleted when the corresponding package is removed
hosting a CI to build system image for different OS doesn’t play well with artifacts system because the package has to be tagged first to trigger the build of the new version then we cannot update the Artifacts.toml file for the corresponding link anymore because that version is already tagged.
the compiled binary is still too large which slows down downloads (but I guess this is not true anymore for 1.8?)
BinaryBuilder only supports cross compilers and Julia cannot cross compile. So as a result we cannot put Julia based applications into Yaggdrial…
These are the main thing I can think of at the moment hope it helps the development of this direction)
Wow, Comonicon.jl is really impressive! You have clearly put a lot of thought into it. Do you have any examples of apps built using it?
After some more thought, I do think you are right that an application should have its own environment (contrary to my earlier suggestion of having a single shared environment). However I do think that an application could be made up of multiple CLI commands (similar to how git is made up of git-xxx commands).
IonCLI is the real world experimental example for building CLI applications (and I think there are a few other CLIs made by others on GitHub too), but it’s currently a bit out of date due to recent changes in Comonicon and I haven’t got time to update it. But this repo should give you a more concrete example of setting up CI for system image and applications (for PC.jl v1 at the moment since I haven’t updated it yet)
there are also simpler examples in the example folder, e.g a fake Pkg CLI interface you can run the FakePkg.comonicon_install with -h in terminal to see the help msg about what it does.
Yes this is possible with the new PackageCompiler v2 I think if multiple entries are defined. And this is related to a feature that I hope to have in the future about CLI plugins, which is a nice thing rust-clap supports.
However I think a big difference from conventional CLI apps built with C/rust/ is when you build your CLI with a system image or applications Julia has a much larger runtime and the runtime actually overlaps between binary applications how to share the runtime is not clear to me yet. Or it would be nice to revive StaticCompiler.jl or whatever is equivalent to get smaller binaries so that copying the runtime libraries is more acceptable. And large runtime also hits latency, I think currently even with system image the startup time of a CLI app is still much slower than rust or C or even go, and similar or slower than python