[ANN] The Ion command line for Julia developers - written in rust!

I’m pleased to announce a rewrite of Ion in rust to provide a much faster UX and much smaller binary. If you haven’t used Ion, please ignore my previous posts and directly read this one!

Everything in Ion is optimized for terminal UX, including its name - ion is the easiest-to-type name that I find not used by other popular CLI tools, and there it is.

This is still at quite an early stage, but I have been using it for a month myself, so I’d like to share this tool with the community. There will be more detailed documentation and a website set up in the future.

TL;DR let me introduce what commands ion provides currently, you can find more detailed information in the help message by running ion help

Installation

Currently only MacOS and Linux are supported because I don’t know how to release Windows binaries, the MacOS and Linux binaries are packed as a tarball in the release CDN, there is only one binary in it, so just download it and put it wherever you like.

Or you can install by building it from a source if you have a rust compiler setup locally, after cloning the repo, you can just run just install and it will install the binary to .local/bin/ folder.

If you want to have shell auto-completion, after downloading ion, run ion completions <shell> to generate the shell completion script, e.g for oh-my-zsh you can copy and paste the following

cd .oh-my-zsh/completions
ion completions zsh |> _ion

In the future, if this is adopted by more people, maybe we can have juliaup ship this or have an ionup for a friendlier installation process.

The forwarded Pkg commands

Ion forwarded Julia’s Pkg command in the terminal, e.g

> ion add
Add dependencies to current environment

Usage: ion add [OPTIONS] [PACKAGE]...

Arguments:
  [PACKAGE]...  The package to add

Options:
  -g, --global  Add the package to the global environment
  -h, --help    Print help

and all Pkg commands will run as equivalent to having julia --project -e "using Pkg; ... by default, and there is a -g --global option to manage global shared environment.

Releasing a new version with Ion

this has been one of the most frequently used features personally in my previous Julia version, and in this new rust version, I have rewritten the whole thing in a much cleaner and modular fashion.

Have you tired of wanting to bump a patch version, but forgetting it’s already bumped in main and only noticing it’s a wrong patch version after the General registry complains to you after 3min? Have you been tired of opening your editor, changing the version in Project.toml manually then opening a browser to summon JuliaRegistrator but having the bot name typed wrong?

With ion release you only need one line to do all these within seconds! e.g

ion release patch

will automatically bump a patch version based on your current version in main and your registered version, if it’s not a continuous version number ion will warn you and ask if you want to continue, if your current version number is already valid, ion will ask if you want to release that instead, and it will summon JuliaRegistrator directly without asking you to open the browser.

release

Custom release pipeline with summon and bump

we now also provide bump and summon commands to allow you to do more customization, e.g managing large mono-repo that contains many other packages or maybe you want to pack up some artifact before summon JuliaRegistrator. You can combine these two commands with just (recommended) or make, like what I do here.

Run a standalone script with dependencies

This is a feature that is supported in many other languages Single-file scripts that download their dependencies - DBohdan.com and this has been a gripe from @vchuravy on slack,

You can write the following in your Julia script

# !/usr/bin/env ion run
#=ion
Example = "0.5"
=#

using Pkg
Pkg.status()

println("hello world")

and ion will parse the #=ion ... block to automatically setup an environment, running this will print

> ion run script.jl
Status `~/.local/bin/env/env-3506815430/Project.toml`
  [7876af07] Example v0.5.3
hello world

You probably want to ask why this is not a Project.toml or Manifest.toml embedded inside the script like Pluto notebook. Like many other similar implementations, we want this part editable and readable since it will directly appear in your script. Thus if you can install a package with a specific version in Julia REPL by only providing the version number and name, you should be typing the same information in your script too! But we have an alternative mode letting you specify UUID, git revision, URL, path, etc, e.g

# !/usr/bin/env ion run
#=ion
Example = {version="0.5", url="https://github.com/Roger-luo/Example.jl"}
=#

using Pkg
Pkg.status()

println("hello world")

On the other hand, there might be cases where we want a script to be never changed, which is something I’m thinking to have a release mode script environment specification that is similar to Pluto notebook that has a complete Manifest.toml and Project.toml inside the script.


As for normal scripts, in that slack thread @davidanthoff mentioned he wants --project to be the default, which is something I advocate too, and I have been using alias jp="julia --project" in my terminal for years. Though this has been a safety concern for julia compiler binary, it is not a concern for a developer tool that only runs locally. So for a script without the #=ion dependencies, ion run is equivalent to the following (and it forwards Julia compiler flags like --threads etc. if you specify it)

command equivalent to
ion run julia --project
ion run script.jl julia --project script.jl

Clone Julia packages

  • Have you gotten annoyed that cloning a Julia package using git ends up in a folder with xxxx.jl by default?
  • Have you been opening a browser, searching the package, copying the package git URL, then cloning the package somewhere?
  • Have you tried to let dev command use your own directory instead of .julia/dev ?
  • Have you cloned a Julia package, ready to contribute to it, but realize you need to fork it and change remote origin to remote upstream and add your own fork?

now ion clone handles all above with just one line! if you try

ion clone Example

It will look for the registered URL and try to clone it, and because you don’t seem to have access to this repo, we will ask if you want to fork it and if you say yes, we will do it for you. No opening browser is needed!

Create a new package with pre-defined templates

  • have you ever typed the same project configuration again and again interactively with PkgTemplates?
  • have you ever typed the wrong option in the interactive mode with PkgTemplates and had to start over entirely?

The ion new command is here to help, we create an entirely new templating system based on PkgTemplates but with serialization in TOML, e.g the following is a project template for small project

name="project"
description = "A project description"

[readme]
[project_file]
[src_dir]
[tests]

and the following is a project template for research packages

name="research"
description = "A research package description"

[project_file]
[readme]
[src_dir]
[documenter]
[license]
[tests]
[repo]
[codecov]
[citation]
[github.ci]
arch = ["x86", "x86_64"]
os = ["ubuntu-latest", "macos-latest", "windows-latest"]

[github.tagbot]
[github.compat_helper]

most importantly you can save your own custom configuration and share it with people! Maybe your company’s internal packages need a custom README template and LICENSE? Create your own template.toml with corresponding components and share it with your colleagues instead of asking them to do it interactively! Check examples in our template registry here!

What’s next?

I’m hoping to have self-update support like juliaup in the future, but I haven’t had the time to work it out, for other features. I’m also thinking about JuliaFormatter and JET integration similar to cargo fmt and cargo clippy, but I haven’t decided on how integration is supported yet.

Last, please feel free to open issues on bug reports, feature requests, or contributing PRs!

56 Likes

Thank you very much for sharing @Roger-luo .
I did not know that I needed this tool, but it sounds really useful. Looking forward to looking into it

Great idea on the user-friendly “templates” system, @Roger-luo !
I’ve never thought that Julia’s project generation system was all that user-friendly, but I really like your solution!

Great work on this, Roger! Your understanding of the intricacies Julia Pkg management system really shines through here, and results in real improvements for user workflows!! :smiley:

4 Likes

Thanks, guys, I’m glad this not only makes me happy but also makes other people happy

4 Likes

Regarding the embedded dependencies, is there any difference from directly invoking Pkg at the top of the script like this?

using Pkg
Pkg.activate(temp=true)
Pkg.add("Example")

Yes it’s basically does this, more precisely it is the TOML version of Pkg.PackageSpec, but if you use the Pkg API, you will need to write

Pkg.add(PackageSpec(url="A"))
Pkg.add(PackageSpec(url="B"))
Pkg.add(PackageSpec(url="C"))

with the ion script dependency

#=ion
A = {url="A"}
B = {url="B"}
C = {url="C"}
#

also, ion will not create a temporary environment, it will create one unique environment for each script path, so no new environment is created, and unlike Pkg.add with temp environment it will not trigger Package/Registry updates if you already have the package installed locally, this will save a lot of start up time.

This is actually one big issue with Pluto where every time you will create a new environment, and thus Pkg will try to install the latest compatible version, most of the time this is not necessary. Ion allows you to manage these script environment manually using ion script, you can either remove the environment or update the environment or start a REPL in this environment.

7 Likes

See also GitHub - tpapp/PkgSkeleton.jl: Generate Julia package skeletons using a simple template system for lightweight package templates. It’s a more direct approach: define a template project directory, laying any files as you wish (or use an existing template), and PkgSkeleton creates projects/packages following this template. No need to learn or use extra configurations!

1 Like

while I appreciate sharing other approaches, I need to clarify that ion new does not require you to learn extra configuration, the template author needs, for users you only need

ion new MyPackage

or

ion new MyPackage -tproject # use project template
ion new MyPackage -tpackage # use package template
ion new MyPackage -tresearch # use research template

plus you don’t have to start REPL and type a bunch of using to generate a package, the goal is to keep workflow within one line :slight_smile:

1 Like

Sorry for the confusion, I meant to suggest using that kind of direct templating system instead of this:

within ion. Clearly, CLI interface has its niche, even for Julia-specific tasks.

Has anyone talked to @fonsp about using your ion package in Pluto.jl? This could potentially solve a lot of the issues they’re having with startup time with Julia 1.9+ …

1 Like

I understand that doing this in a simple way has the benefit of being less work and easier to maintain but to minimize users’ work in setting up a package as a dev tool we must do as much as we can.

I might be too brief on introducing the templating engine in Ion - it’s a new implementation but old design. I learned this plugin design from PkgTemplate and re-implemented it in rust with serialization via serde. And it uses the popular templating language handlebars, thus a lot more flexible than what PkgSkeleton supports.

Making people themselves edit their project works for single developers but doesn’t work for a team - I want all packages created inside an organization to be consistent on licensing, README, CI, etc. which I think is quite a common thing within a company or just a team.

People will make mistakes, or maybe just want to be lazy on this and revisit it later. So asking them to edit things manually is never gonna make things consistent and sounds terrible. At least I personally prefer PkgTemplates over PkgSkeleton for this reason (I’m aware of these existing solutions while designing mine actually). The only thing I dislike in PkgTemplates is the lack of serialization, I actually had a PR for PkgTemplates but no one had time to review it back then.

So in conclusion, I don’t think I want to remove features from our template system in any case.

1 Like

No, I haven’t, but yeah I have been reading that long thread on discourse, and I did have a short discussion with Fons one or two years ago about script dependencies. But I think we should let people try my current design more before we spread it across the ecosystem.

there is a few known issues that I haven’t resolved, although we currently do not try to install new versions if there is a compatible version already exists in the generated Manifest.toml, we are not checking this compatibility very carefully, the version range spec used is following node semver instead of Julia’s version range spec, thus there are some inconsistency in the syntax. We need a rust crate for Julia semver to fully resolve this, which I’m still working on and it’s gonna take a while.

I’m not sure if there are other issues, so I’d rather let this float around a bit and then decide what to do. On the other hand, besides potential immaturity, asking Pluto to use this design will require shipping a binary which might be tricky, I haven’t figured out how to build Windows binaries… If someone could help me.

Maybe I’m missing something, but I don’t see why this tool written in Rust is preferable to having everything in Julia?

it’s probably easier for the package developer in the first place, since rust has a fairly mature ecosystem of CLI crates that makes it easy to build off of. i.e, these:

Even though I love Julia for scientific computing, given its latency issues, I would not consider it appropriate for command line tools. Rust seems pretty good for that. In fact, I’d love to see more Rust tools in the Julia ecosystem, e.g. linters

5 Likes

So I’ve been reading through your codebase and familiarizing myself with it so I can help with further development. So, are you ready & open to having contributors at this point (provided they follow Ion’s current style)?

If so, let me know what you would be most interested in having a contributor work on.

A couple of ideas I had in mind for contributing are:

I think you mentioned that there were a couple of features from Maurice that you liked - if you want me to implement any of them, please let me know which features you want, and I’ll take care of implementing them for you.

I could also look into Windows compatibility and/or producing Windows binaries. I make no promises on this one, since that’s not a task I’ve done before, but I’ll see what I can do. This was something I planned to get around to doing anyhow for my own needs, so it’ll be a good learning experience.

Or something else - whatever suits you. I’d like to contribute something, since I think I’m going to take you up on your suggestion and re-use part of your code for my future GUI experiments (you’ll be credited, of course). I really like your new templating system!

I was wondering, though: do you plan on keeping one centralized repo for templates…? Do you think that repo might get cluttered over time, if there’s community (and corporate) adoption?

Also, would you be interested in adding a command to display what’s in a template from the Ion CLI (ie. being able to print out the template to the terminal without needing to visit the repo if you forget)?

I think that might help with onboarding of new users with Ion, like with understanding the purpose of the templates that are already installable with Ion…

Thoughts…?

3 Likes

I agree. Which is why I think using Julia in a “scripting” way is a mistake. Just fire it up and then keep working. In this way the functionality described here could be easily replicated fully in Julia (as it really already is, just consider the various packages already mentioned in this thread). Summary: I don’t see why to get excited about having to deal with Rust.

Whether you like it or not, there are many things that have to use Julia as scripts, such as DevOps, and HPC jobs simply because when there is no interactive process available, it’s impossible to keep a REPL in those cases. To find an example, just look at our CI in the ecosystem, and the registry’s tests, there are just too many examples. To interact with other programs, scripts are the simplest way of doing it. Not to mention that Revise does not always work.

Wow, how easy that is? Does PkgTemplate support serialization? is there any Julia program that allows you to release a package directly? How many keystrokes do you need to type just to release a package? Prove this by developing a Julia version of such functionality like me, then talk. Talk is cheap and saying something people spend a few thousand lines of code to provide is easy is not cool.

7 Likes

Yes definitely! The code currently lacks of documentation, so please feel free to ask me directly if anything looks confusing.

I guess currently the best thing to start with is resolving the CI such as Windows support, compared to what juliaup provides, Ion only has poorly 3 platforms supported… the main blocker for me is I just don’t have any knowledge about developing windows CLIs at all, so if you know windows executables like how things can be packed that would be a nice thing to add.

No, I hope templates can be customized and used for private purposes, which is one of the reasons why I develop this template engine - I want to use it for a company’s internal packages which may have private CI, README that contains the company logo, etc. But this is a good question that I haven’t put much thought into, and I currently just use a git-repo as a lazy solution.

Yes, I think this is a good idea! There is a template subcommand that can list available template names and descriptions, maybe we can have something like ion template inspect <name> to print the TOML!

I think the best contribution would be something that is useful for yourself, and I’m open to anything that is useful for making DevOps work easier.

And thanks for your kindly offer, looking forward to it!

1 Like