# Tooling for Julia command-line scripts
**URL:** https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915
**Category:** Package Management
**Tags:** cli
**Created:** [January 2, 2022, 5:41am UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915 "2022-01-02T05:41:30Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [January 2, 2022, 5:41am UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915/1 "2022-01-02T05:41:30Z")
Prompted by [Jeremy Howard’s reply](https://twitter.com/jeremyphoward/status/1477213680970305537) to @viralbshah’s [question about 2022 goals for Julia](https://twitter.com/Viral_B_Shah/status/1477211793688698881), one thing I would like to figure out how to make it easier to write and deploy command-line scripts written in Julia.
I think the key things to figure out are:
1. How should such a script be packaged? The obvious answer I can think of is as a regular Julia package, with a `bin` subdirectory containing the necessary scripts. This pattern is already used by some packages, but there are certain conventions that should be established, e.g.:
2. How should a user manage such packages? Ideally, installation should be a single step, and result with the script name appearing in the users `PATH`, and should have a simple mechanism for upgrading or removing it. There have been a couple of different approaches either used or suggested:
3. How could this system integrate with (a) PackageCompiler.jl (e.g. to compile a specific script or system image), and (b) system package managers (e.g. so that a script could be installed via `apt`, `yum` or `homebrew`)?
Any thoughts or ideas on how we might improve things?
---
### Author:  [@akb](https://discourse.julialang.org/u/akb)
#### Post date: [January 2, 2022, 8:34am UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915/2 "2022-01-02T08:34:06Z")
Unix scripts may have shebang `#!/usr/bin/env julia`.
### CLI tools using a system’s existing julia installation
Declaratively define the set of installed applications in `~/.julia/config/applications.toml`.
```julia-auto
# Define a project for linting that has access to StaticLint
# and an optional dependency of it.
[applications.linting]
juliapath = "/usr/local/julia-1.7.1/bin/julia"
[applications.linting.deps]
StaticLint = "b3cc710f-9c33-5bdb-a03d-a94903873e97"
AnOptionalStaticLintHelper = "571b7581-3216-41df-b45e-703c535b21e8"
# Avoid conflicting binary names by specifying links.
[applications.linting.link.mapping]
"/StaticLint/bin/staticlint" = "jlint-classic"
"/AnOptionalStaticLintHelper/bin/staticlint" = "jlint-helper"
```
Installation creates an `applications_manifest.toml`.
```bash
% julia
Pkg> app install --edit-profile
Installed:
~/.julia/bin/jlint-classic -> ~/.julia/packages/StaticLint/ABCDE/bin/staticlint
~/.julia/bin/jlint-helper -> ~/.julia/packages/AnOptionalStaticLintHelper/FGHIJ/bin/staticlint
The directory ~/.julia/bin was added to your shell profile.
The next time you log in the programs should be available.
^D
% ls -l ~/.julia/bin/
~/.julia/bin/jlint-classic -> ~/.julia/packages/StaticLint/ABCDE/bin/staticlint
~/.julia/bin/jlint-helper -> ~/.julia/packages/AnOptionalStaticLintHelper/FGHIJ/bin/staticlint
# Next session, the PATH is changed.
% echo $PATH
~/.julia/bin/:~/.local/bin/:...
```
Inspired by [venvs](https://github.com/Julian/venvs/), [pipx](https://github.com/pypa/pipx).
### Standalone applications
Using a system’s Julia compiler can [cause issues](https://xkcd.com/1987/) that static binaries don’t have. Unlike ~/.julia/packages, the set of installed julias isn’t controlled by the Pkg team. How small can Julia’s standalone sans-llvm binaries be?
---
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [January 2, 2022, 8:50am UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915/3 "2022-01-02T08:50:57Z")
I think although it’s not perfect but under current package system we can already do something. Here’s what [Comonicon](https://comonicon.org/stable/) does:
1. an `install` function is provided to install the CLI scripts
2. all CLI scripts live in `~/.julia/bin` is tied to a specific environment created in corresponding package’s scratch space (using Scratch.jl), this is because:
- in order to make compile cache work one will need to define most of the implementation in a project module (maybe in the future we can also do some compile cache for specific type of script file instead of project modules only?) this is quite crucial for a non-trivial CLI to work with low latency (yes after many times of trying I find supporting CLI scripts directly with custom compile cache doesn’t help much for complicated scripts, the best way is still to write project modules, so this feature is removed from Comonicon in recent versions)
- to make system image work the CLI needs to be tied with a specific Manifest.toml
- the scratch space can be properly removed after the package is removed
- different CLI shouldn’t share one environment because they may depend on different versions and they shouldn’t use the package environment because we want to fix the dependencies and use other package to generate related method cache (e.g to support CLI plugin) (a mutable Manifest.toml file instead of immutable Project.toml) once it’s installed.
1. A config file `Comonicon.toml` is used to define the default behavior of installation and building, e.g compile options for starting the CLI and for compiling using PackageCompiler (nthreads, compile=min, etc)
2. The CLI package author can choose to run installation by default via putting the `install` function in `deps/build.jl` (This is not ideal tho)
3. The installed scripts however is not a julia script but a small shell script + a small julia script, this is because to make the latency reasonable one won’t want to have a lot of code in a single Julia script. And the bash script is used to config the start up option (the environment path, system image path etc)
4. The shell autocomplete are generated if the default shell is supported (currently only ZSH) then the completion scripts are installed at `~/julia/completions`
5. If the author choose to install with system image (either via local build or download prebuilt system image binary from GitHub release CDN), the system image will be installed into the scratch space of corresponding package too. This makes sure when one updates or remove the package system image can get updated.
6. Package author can also choose to build applications which compiles everything into a binary with a bunch of shared libraries (via `PackageCompiler.create_app`) and pack them into a tarball. This workflow is also supported with `install` function (itself actually is also a CLI). So after compile the whole thing then upload it to GitHub release CDN nothing is tied to Pkg or julia anymore it’s completely standalone (so can be used as an artifact in principle)
So a few things I’m not happy with the current mechanism Comonicon uses as a workaround due to limitations of Pkg
- since there’s no mechanism for installing things that has a build process of system image, users will not see a progress bar of building or downloading system image if the author choose to use a system image for the CLI (because all the info from `build.jl` will be redirected, unless with verbose option)
- the installed scripts in `.julia/bin` and `.julia/completions` will not be deleted when the corresponding package is removed
- hosting a CI to build system image for different OS doesn’t play well with artifacts system because the package has to be tagged first to trigger the build of the new version then we cannot update the `Artifacts.toml` file for the corresponding link anymore because that version is already tagged.
- the compiled binary is still too large which slows down downloads (but I guess this is not true anymore for 1.8?)
- BinaryBuilder only supports cross compilers and Julia cannot cross compile. So as a result we cannot put Julia based applications into Yaggdrial…
These are the main thing I can think of at the moment hope it helps the development of this direction)
---
### Author:  [@simonbyrne](https://discourse.julialang.org/u/simonbyrne)
#### Post date: [January 2, 2022, 7:02pm UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915/4 "2022-01-02T19:02:56Z")
Wow, Comonicon.jl is really impressive! You have clearly put a lot of thought into it. Do you have any examples of apps built using it?
After some more thought, I do think you are right that an application should have its own environment (contrary to my earlier suggestion of having a single shared environment). However I do think that an application could be made up of multiple CLI commands (similar to how `git` is made up of `git-xxx` commands).
---
### Author:  [@Roger-luo](https://discourse.julialang.org/u/Roger-luo)
#### Post date: [January 2, 2022, 7:41pm UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915/5 "2022-01-02T19:41:59Z")
IonCLI is the real world experimental example for building CLI applications (and I think there are a few other CLIs made by others on GitHub too), but it’s currently a bit out of date due to recent changes in Comonicon and I haven’t got time to update it. But this repo should give you a more concrete example of setting up CI for system image and applications (for PC.jl v1 at the moment since I haven’t updated it yet)
> [@\[ANN\] IonCLI: your package manager and more in terminal](https://discourse.julialang.org/t/ann-ioncli-your-package-manager-and-more-in-terminal/44946):
>
> Although I still have a lot feature want to add to it, I think it’s time to announce this and let people try it since I have been making whoever I know to try it in past few weeks. Thus I’m announcing a new CLI package [IonCLI.jl](https://github.com/Roger-luo/IonCLI.jl). It’s still at an early stage, even I have tested it myself in daily usage for a few weeks, I still expect it to have some glitches, and please file an issue to help improve it if you hit any. IonCLI is a CLI command made for managing Julia packages: most of its command…
there are also simpler examples in the example folder, e.g a fake Pkg CLI interface you can run the `FakePkg.comonicon_install` with `-h` in terminal to see the help msg about what it does.
>
>
> //github.com/comonicon/Comonicon.jl/tree/main/example/FakePkg
I think recently this CLI app is also using it for binaries: [GitHub - KwatMDPhD/Clean.jl: Command-line program for cleaning 🛀](https://github.com/KwatMDPhD/Clean.jl)
> [@simonbyrne](#):
>
> However I do think that an application could be made up of multiple CLI commands (similar to how `git` is made up of `git-xxx` commands).
Yes this is possible with the new PackageCompiler v2 I think if multiple entries are defined. And this is related to a feature that I hope to have in the future about CLI plugins, which is a nice thing rust-clap supports.
However I think a big difference from conventional CLI apps built with C/rust/ is when you build your CLI with a system image or applications Julia has a much larger runtime and the runtime actually overlaps between binary applications how to share the runtime is not clear to me yet. Or it would be nice to revive StaticCompiler.jl or whatever is equivalent to get smaller binaries so that copying the runtime libraries is more acceptable. And large runtime also hits latency, I think currently even with system image the startup time of a CLI app is still much slower than rust or C or even go, and similar or slower than python
---
### Author:  [@Lester7458](https://discourse.julialang.org/u/Lester7458)
#### Post date: [January 8, 2022, 9:01am UTC](https://discourse.julialang.org/t/tooling-for-julia-command-line-scripts/73915/6 "2022-01-08T09:01:20Z")
Thanks for sharing this information. It was very useful[.](https://www.mymilestonecard.biz/)