Towards standalone (package/project/etc.) management app

Tooling
1

Hi all!

I have been thinking, building, and experimenting standalone manager app implemented in pure Julia (well there are about 50 loc of C in PackageCompiler) in the past few months which is the IonCLI. I’d like to share my findings and start a discussion on the standalone package manager built in Julia and for Julia. And maybe draw some attention from Pkg team as well! (@StefanKarpinski) Since it won’t be possible without some more changes on Pkg side in the future and this demo is not stable enough to use for production at all.

A glance at what I have implemented as a demo

https://github.com/Roger-luo/IonCLI.jl/blob/master/demo.gif?raw=true(image larger than 4 MB)

it’s a standalone application built with PackageCompiler, so you don’t need to install julia to use it

Background

By standalone package manager, I mean a package manager does not depend on the Julia compiler one uses or will use. It is an application built by PackageCompiler and can be download and use without installing a Julia compiler.

demo

In other programming languages, this usually refers to things like rust’s cargo, nodejs’s npm.

Advantages

Although the experience of Pkg in REPL is almost perfect, there are still quite a few advantages of having such manager apps:

Not a Stdlib - more features & fast evolve

I guess, as we all agree, code in stdlibs are half dead.

Since it is a standalone application, it won’t and can’t be a stdlib (or we have to ship it with a complete Julia compiler), this means we can use whatever already exists in the package ecosystem and just depend on them. This means you can support project creation via PkgTemplates, colorful and fast CLIs via Comonicon, and implement a lot other features in a package instead commit them into a centralized stdlib. And increase the functionality without worrying about increasing the dependencies of Julia binary.

Upgrade & Manage Julia Compiler Binaries Seamlessly

I believe lots of people have experienced this: when a new Julia compiler version comes out, you will need to uninstall the previous version manually to do the upgrade, or you will have two or multiple versions of Julia even you don’t need them. And if you are someone like me living on Julia’s master branch, switching/updating julia versions can be quite often.

Moreover, one has to dig into the .julia/env folder to copy the old environment to the new environment. And because of firewalls, one may need to download the binary via proxies, from a mirror, etc. This might be fine for power users of Julia, since we know a lot of details of how things work, but it is cumbersome for most people, which is why there have been efforts like jill.py, jill, etc.

This feature would be hard to implement within Pkg since Pkg is shipped with the Julia compiler. It can be hard to delete and modify yourself. But it is very simple to implement using Julia as a standalone application.

Beautiful CLI and GUI

As a standalone application, we can easily provide a nice CLI and GUI - this would make the ecosystem more accessible to all levels of users. I don’t have a demo for GUIs (but hopefully some point in the future)

Ideally, like rust’s cargo, it can be a small binary that does not depend on the compiler at all, but I find the size of the current build is only about 70~90MB, which is still acceptable I believe.

Closely optimized workflow for CLI/GUI based programming

Julia has an excellent REPL experience already, but due to various reasons (largely about latencies, but also because of the lack of tooling), I have to say the experience of CLIs was not great. But I think with the progress of various packages, now it’s time to make this great and smooth.

As I demonstrate above, with a CLI one can simplify the workflow in the terminal a lot, without opening REPL at all, given in a lot of cases, one just wants to do very simple things, like manage packages, or create a package, or search a package, etc. It’s more convenient to have an entry for this type of task directly from the terminal or potentially a GUI.

Moreover, as a lot of programmers with other programming language experience using CLIs, I think this workflow would much more familiar to a larger population of programmers (since CLIs are what people have been using since the born of Unix), I believe this would help in closing the gap for beginners.

The ability to manage other Binaries with BB and JLLs

As @vchuravy pointed out on Twitter, we can actually manage other binaries like git, fpmpeg via JLLs directly, so this could potentially be something more powerful and a replacement of things like brew with better cross compilation ability through BinaryBuilder.

Looking Forward

I think 1 year ago, it’s not imaginable to have such a CLI application implemented in pure Julia. But as the compiler keeps progressing (and hopefully someday when the static compilation pipeline becomes mature, so we can have much smaller size of binary) I think the community can already think about this and target this.

31 Likes
2

I cannot comment on the technical advantages of this approach, but I’ll share my limited experience with standalone package manager.
But first, a question: is IonCLI a proposal for replacing the package manager or an additional tool that packs additional functionality but still depends on the package manager?

When I decided to switch away from Matlab, not being a CS person, not being used to cli and working on Windows, I wanted to choose between Python and Julia. One of the reasons I chose Julia was the package manager built into the REPL, the ease of installing packages and activating environments.

I gave Python a try and went mad with pip being incompatible with whatever packages and all sorts of errors I couldn’t understand. This gets worse as different distributions have a slightly different workflow: conda, standalone install, pycharm. I found myself constantly switching between windows during installation of packages and creating environments, trying to understand which console to use or check for errors.

As a basic user I find it much more convenient to have this functionality available in the language itself, one key press away, than to switch to another window and do the same thing and worry about compatibility and all the usability quirks. I think the powerful REPL with built in package manager is one of the strong points of Julia.

Other languages don’t have such a powerful REPL and an external tool is their way of managing packages, but why steer this away from Julia?(Racket also does thia with raco and I can’t understand why; maybe because it was faster to do it that way?)

I’m sure the technical reasons you stated are sound and please don’t take my comment too harsh because as you can see I’m really biased against Python and pip.

5 Likes
3

The later. As mentioned and demonstrated above it’s based on many existing Julia packages and Pkg itself.

It’s fine if you want to stick to REPL workflow but it’s not fine if we don’t have an option for people who like CLIs with a very smooth CLI workflow.

The point of this post is about providing an alternative to make things easier not replacing things. And most importantly a way to download and upgrade Julia binaries.

And I’d argue that at least in the workflow of my demo you have less keys to press since at least you don’t need to start REPL to just install packages :wink:

the point is to use one’s favorite workflow

And as I mentioned you won’t be able to manage your Julia compiler binary with REPL by implementation.

IMHO python package manager pip is just a bad example and it’s not standalone. Usually such package manager requires the language to be able to generate binaries. Python community also wants to switch to something more modern but that’s another story. I’m sorry you start your first CLI with pip and it ruined your experience. As I have mentioned above I’m referring to things like rust cargo etc.

And from your description I assume you are using windows I don’t think using CLI in windows is a good idea unless you are familiar with PS. A GUI for Pkg makes more sense to me on windows which I think will eventually goes into Pluto.

3 Likes
4

Thank you for the clarifications! Your first answer cleared my doubts and I agree that for the more CLI inclined/developer this would be a good addition.

5

I think this is a wonderful idea! As an intermediate user I too am very happy with Pkg but love the idea of a Julia-based standalone that could bootstrap me into the most recent Julia version and I feel it might help my Julia-naive friends get started a little more smoothly as well. There might be details I’m not considering, but on first pass it seems like a great option to have, thank you for creating this even in prototype form!

1 Like
6

Thanks! I think stand-alone applications should eventually give Julia a one-click experience on managing all the package related operations (not just adding dependencies, but also creating packages, releasing, building, serving and more)

I created this post to attract more people think in this direction.

1 Like
7

I think 99% of the problems with pip stem from how old Python is as a language. When it was being developed were didn’t have the idea of a Package Manger as an essential piece of a language and is ecosystem along with the idea of a single official source which everyone used to share code as packages. Hence the whole thing was an afterthought and all the pain of distutils, setuptools, the various approaches to virtual environments etc followed as the community tried different things and later had to support things which didn’t work.

Ruby would be the perfect contrast in terms of a language that was developed when it was apparent from the beginning that this was something you wanted to think about from the get go.

3 Likes
8

I don’t think age is an excuse. Ruby is almost as old as Python — 25 years versus 29 years. Perl is even older and was the original language that crushed it at package management. Python core devs just don’t seem to care about package management and consider it to be someone else’s problem. I can sympathize, since it is a major pain, but Julia is good at package management because Julia devs consider it an important feature and a responsibility.

18 Likes
9

@StefanKarpinski @hal9zillion I agree with you both - each has its reasons. But no matter what the reason for pip is, pip is not a good reference at all, let’s just ignore it. And the fact that pip is not great does not mean CLIs are awful, @Iulian.Cioarca really just had the bad luck to use pip as his/her first CLI. I didn’t post this to talk about pip at all - it’s not even in my consideration of comparison.

And I think there are many many CLI lovers who would like to use a CLI - it does make things more convenient. And it does get you rid of a lot julia -e and even more complicated commands in your terminal scripts, GitHub actions, and simplify the general workflow in the terminal. Because this is why I made IonCLI - I made it for myself in the first place.

Thus, I don’t think CLI and GUI for Pkg is something we should ignore due to Python or other language’s failure, because we can do much better. With the on-going static compilation work that @jpsamaroo is working on, I believe Julia can (eventually) generate awesome CLIs like those rust CLIs as well. And even with current PackageCompiler, one can already make a low latency CLI with some careful programming on relocatibility. Thus a CLI for Pkg made in pure Julia could be the first application to think about.

However, I think one step further towards this is to make Pkg more relocatable, which for example:

I’m not a core dev in Pkg which means I’m not familiar with its development plan, so this might require @StefanKarpinski’s and others’ opinions and thoughts. Since this would be an extra assumption when developing Pkg’s code.

I think relocatibilty would in general help all standalone applications that may potentially calls Pkg not just the Pkg CLI. Not just for Pkg CLI (IonCLI) itself, there can be a more general use case: one can use Julia’s Pkg for hot loading plugins in general as well (with a julia runtime shipped with), which traditionally requires the two language solution like C++/Lua on this. This is also something I just experimented with in IonCLI.

5 Likes