[ANN] The Ion command line for Julia developers - written in rust!

[PetrKryslUCSD]

Look, all I am saying is: Julia already has package and project management. It may not be perfect, but
if there are pain points, I would much rather see some effort put into a Julia-native solution than a bolt-on in another language.

[Roger-luo]

I think it’s pretty clearly explained above and other people have also explained why rust is much better in writing these devop tools, but since you keep saying this:

• I have been implementing the same functionality in pure Julia since 2020 (that’s why I was asking you to open that link and clearly you didn’t read them) and if you want to use the same functionality from REPL that’s also what you can use, and yet you ignore these replies
• I have tried to improve existing packages and it didn’t work out well
• I created one of the best CLI generator in Julia for this, but it’s still limited by Julia compiler, we don’t have a solution for building standalone DevOps tools with Julia

Now if you want to prove that I’m wrong write a better prototype than what I did, again talk is cheap show me your code!

[viralbshah]

@PetrKryslUCSD

Look, all I am saying is: Julia already has package and project management. It may not be perfect, but
if there are pain points, I would much rather see some effort put into a Julia-native solution than a bolt-on in another language.

I don’t fully agree even though I understand the sentiment. Yes, it would be nice for folks to improve Julia’s built-in tools, but it is also perfectly fine to explore new things, and sometimes it is better to start afresh. I understand that this leads to more fragmentation, but sometimes new ideas are better off being off the beaten path, where they can mature quickly and overcome inertia. If successful, they can be brought back into the main line, or remain an alternative, or be a good experiment.

[orco]

HI @Roger-luo ,
Just out of curiosity: did you choose to (re)-implement IonCLI.jl in Rust because you already knew Rust, or did you choose Rust because of its good reputation, and learned on the fly the language? I’m curious of the latter, whether it was easier for you to learn Rust than Julia,. or the other way around… Of course your prior programmming languages background might help you, but I’d like your opinion.
Thanks, and great work!

[Roger-luo]

I chose to re-implement IonCLI and its REPL version IonBase because

• it’s very very hard to cross-compile and ship system images generated by Julia compiler at the moment
• the latency of the CLI/REPL is too bad compared to what’s available potentially
• I need something that does not depends on libjulia.so so that I don’t need to update and re-compile the system image every time I have a new Julia.

Even though there is progress on StaticComipler, PackageCompiler, and removing Julia runtime after compilation, the above issues are still far from resolved in the near term - the main use case for Julia is not on DevOp tools after all.

Thus, I need to find a language that solves the above problem for me, I considered Go, Python, and C++, but just dislike the language design and tooling there. Plus there is already a nice juliaup existing in the community.

---

I’m not sure what is counted as already known rust. I know a bit of rust before I started writing ion because I was trying to fix an issue in juliaup, and I have been working on bringing rust and other FP language features into Julia (MLStyle, Expronicon). And I was mainly a C++ programmer before I start working on Julia.

I learned most of the things I need on the fly (like what package to use, and what code style should be enforced, I think these things take more time), it has some new concepts but not as hard as people complained about on the internet. I learned these by reading the source code of cargo, and by contributing back to the rust community. So if you want to learn rust I think you should just go ahead without worrying about anything. And also make sure you actually need this language for a certain concrete project.

[Palli]

@Roger-luo

wants --project to be the default, which is something I advocate too

That seems like a breaking change, but isn’t with a different tool like ion (or the alias).

This is a feature that is supported in many other languages: Single-file scripts that download their dependencies. The lack of this feature has been a gripe from users in the Julia community; Ion solves this pain point with its new script system.

One of the dependencies is Julia itself, which I think isn’t downloaded (but could be). Your tool ion and juliaup could be separate for now (or merged eventually, so downloading juliaup would install both tools). The question is which Julia you would want downloaded or selected, should it be listed in the #=ion comment (if not which then?).

This would also work on Windows:

ion run script.jl	julia --project script.jl

But ideally on Windows (and elsewere), you would want to invoke with:

script.jl

but then it would invoke julia, not ion first, since on Windows the file ending controls this. We could have .jls for scripts… or .ion? Additionally we seem to need .jlw (similar to .pyw on Windows plus .py), not just .jl (see other thread on it), so that’s non-ideal if it means 2*2=4 extensions… Possibly we can get away with just one new extension (for now).

FYI: It’s actually possible to use .exe for binary files that are also shell files, and portable between Windows and Linux etc… I could foresee such files where you have a script with a short ugly preamble that downloads julia, ion whatever for you, then ends with the script, in one file (or if you wish with julia runtime in there, or (only) your compiled Julia code, still cross-platform):

https://justine.lol/ape.html

[Palli]

Since at least two binaries are needed on Windows for Julia (for console and non-console .exe distinction there, unlike in Linux), I’m ok with one of them being ion:

@PetrKryslUCSD

I would much rather see some effort put into a Julia-native solution than a bolt-on in another language.

Rust is just an implementation detail (that I can support).

More importantly, since ion is mainly for scripts, I suggest running julia with the lowest optimisation possible, -O0 (if not --compile=min, which seems like going too far as a default). Since ion is a rather new project (in the reimplementation), I suggest doing that change as soon as possible. Since arguments can be passed on to julia from ion, users could always override it in the CLI themselves (and maybe in the script also?).

For those clicking on the script in Windows (or Linux), it’s best to have sensible defaults, and it would be a hassle to override to -O0 (if that is not chosen as default), I believe it can though be done with a shortcut file, but then you need two files… Also since the scripts are most likely meant to run silently in the background (also useful if you want to pop up a GUI), I suggest the default isn’t a console .exe. The file could always be run with julia (I’m assuming julia is a console types of .exe).

[Roger-luo]

@Palli

One of the dependencies is Julia itself, which I think isn’t downloaded (but could be). Your tool ion and juliaup could be separate for now (or merged eventually, so downloading juliaup would install both tools). The question is which Julia you would want downloaded or selected, should it be listed in the #=ion comment (if not which then?).

this is a good question, I must admit I haven’t thought much,

for compile options, you can write things like

# !/usr/bin/env ion run --threads=4 --sysimage=sysimage.so --
#=ion
Example = "0.5"
=#

using Pkg
Pkg.status()

println("hello world")

ion currently allows you to set a path/shell variable to julia in its config file (the command is wrapped into CLI yet tho), by default it just calls julia, if you have juliaup installed, this is managed by juliaup and ion will not touch it.

But allowing a special field in specifying julia compat version might be a good idea, since a given script could only work on a later julia version only.

@Palli

But ideally on Windows (and elsewere), you would want to invoke with:

This is out of my knowledge, as I have zero experiments developing things with Windows, e.g does Windows allow a shebang-like thing somewhere? IIUC, you are saying Windows can only dispatch on file extensions instead of shebang?

@Palli

More importantly, since ion is mainly for scripts, I suggest running julia with the lowest optimisation possible, -O0 (if not --compile=min, which seems like going too far as a default). Since ion is a rather new project (in the reimplementation), I suggest doing that change as soon as possible. Since arguments can be passed on to julia from ion, users could always override it in the CLI themselves (and maybe in the script also?).

I haven’t forwarded these two options but e.g when forwarding Pkg commands to Julia, ion indeed use --compile=min. The reason why I’m not doing this by default is that currently most of the packages do not specify their module-level optimize and compile options, which as a result when we turn --compile=min globally, the packages get called will also be run with --compile=min IIRC. I’m not sure if there is a way only executing the script with --compile=min but with all packages, it depends on getting -O3 with compilation.

One use case for myself is running a complicated simulation with a CLI interface (so that I can submit a lot of parameters to slurm) on HPC, I’m willing to pay some overhead to make my actual execution fast. If it’s about creating a more permanent tool using a Julia script, maybe just having a shebang like

# !/usr/bin/env ion run --threads=4 --sysimage=sysimage.so --

is easier? despite whether it works on windows or not.

@Palli

For those clicking on the script in Windows (or Linux), it’s best to have sensible defaults, and it would be a hassle to override to -O0 (if that is not chosen as default), I believe it can though be done with a shortcut file, but then you need two files… Also since the scripts are most likely meant to run silently in the background (also useful if you want to pop up a GUI), I suggest the default isn’t a console .exe. The file could always be run with julia (I’m assuming julia is a console types of .exe).

I need to admit again that I don’t know much about Windows, so I didn’t consider the use case of clicking a script at all. Perhaps I could read some basic materials about how Windows manages a clickable script first if anyone has any.