Juliaup: a modest suggestion

Internals & Design
22

A way to look at this is that juliaup is the common interface between the “inside” and “outside”. We’ve gotten to the point where juliaup itself is easy and reliable to install and it can reliably install and manage various Julia versions. So far so good—that’s the “inside”. There’s more work to be done on the inside: mostly feature development, but some bug fixes too. But it’s good enough to be the way to install Julia (and if we don’t make that jump, it will never get there). If there are specific issues people have with how juliaup installs and manages Julia versions, they should be discussed in juliaup GitHub issues. Don’t like that juliaup uses a hidden directory on Windows? Open an issue about it. The answer is not to reject juliaup entirely because of one thing you don’t like.

The next frontier is working on the outside: building various native ways to install and interact with juliaup and then letting juliaup be the one thing that deals with the actual installation and management. Right now the juliaup experience is actually most native on Windows—it leverages the standard way to install and update apps. That seems to be blocked/disallowed in a lot of places, which means we need alternatives. Maybe the best option would be to have a single Wizard installer that everyone can use that installs via winget if available and if that doesn’t exist or fails, tries installing another way.

The installation process on other platforms is pretty simple but piping commands from curl into a shell is a little sketchy (it’s safe as long as the URL is HTTPS and you trust the server, but still). It may be better to have more platform-native installation mechanisms. On MacOS that would be a dmg that installs a JuliaUp app with a simple GUI that lets you manage Julia versions and can launch Julia in a terminal. But (a) someone has to build and maintain that, and (b) that’s personally not an experience I actually want, and I suspect I’m not alone. I prefer the pure terminal experience.

On Linux, the native install experience would be for each distro to have a package to install juliaup. Since juliaup has zero dependencies by design, that should be very easy, but it does need to be done on each platform.

One risk of having more native ways to install juliaup is that the curl method will get less testing and might not work as smoothly, but it’s already works pretty well and maybe doesn’t need more development.

17 Likes
23

Reached the point where we are in “violent agreement.” Looks like the right path forward. Second time I have tried to close the worm-can I opened.

6 Likes
24

Glad we’re in agreement! I wasn’t sure if we were, but this was a good opportunity to lay out what I see as the “juliaup vision” and why we want to converge on it as the one true way to install. Yes, it can do more than what some people need, but it means that when someone does suddenly find themselves needing to do something more complex, they don’t abruptly have to completely switch installation mechanisms.

6 Likes
25

I think this might be where the disconnect is - it’s hard to speak for “most Windows users” of course, but (assuming you are referring to the Windows Store here) in my experience this isn’t the standard way at all. I’ve been using Windows since 3.11 and the time honoured way of installing “apps” (which in itself isn’t a very Windows-y term) is download an exe installer onto your desktop, double click it, click “Continue” as often as required and end up with a link to the installed program on your desktop (and a Start menu entry).

The Windows store was only invented in its current form for Win10 afaik and came more from Microsofts Windows mobile efforts and other platforms so sat somewhat oddly with standard Desktop computer Windows for a long time. So maybe somewhat ironically I would say that the move from just downloading the Julia installer as an exe from julialang.org/downloads to juliaup made things less “standard” for Windows users.

(You also said “standard way to install and update apps”, which tbh I’m not sure there is a standard way of updating apps on Windows…)

12 Likes
26

I will add the reason why I personally dislike the juliaup method as a Windows user, and that is corporate IT policy. The Windows App store and all of its components are blocked, which means that neither the recommended method of acquiring juliaup works, including the manual download MSIX installer method. The final option for the MSI installer is also blocked by IT policy where the .msi extension is blocked from download by the firewall.

In short, the only method that works is a single user install with the .exe installer.

5 Likes
27

I hope your proposed fixes make all of this go away:

info: downloading installer
Welcome to Julia!

This will download and install the official Julia Language distribution
and its version manager Juliaup.

Juliaup will be installed into the Juliaup home directory, located at:

  /Users/lewis/.juliaup

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /Users/lewis/.juliaup/bin

This path will then be added to your PATH environment variable by
modifying the profile files located at:

  /Users/lewis/.zshrc

Julia will look for a new version of Juliaup itself every 1440 minutes when you start julia.

You can uninstall at any time with juliaup self uninstall and these
changes will be reverted.

✔ Do you want to install with these default configuration choices? · Customize installation

✔ Enter the folder where you want to install Juliaup · /Users/lewis/.juliaup
? Do you want to add the Julia binaries to your PATH by manipulating various she✔ Do you want to add the Julia binaries to your PATH by manipulating various shell startup scripts? · yes
✔ Do you want to add channel specific symlinks? · no
? Enter minutes between check for new version at julia startup, use 0 to disable✔ Enter minutes between check for new version at julia startup, use 0 to disable · 1440
? Enter minutes between check for new version by a background task, use 0 to dis✔ Enter minutes between check for new version by a background task, use 0 to disable · 0
Juliaup will be installed into the Juliaup home directory, located at:

  /Users/lewis/.juliaup

The julia, juliaup and other commands will be added to
Juliaup's bin directory, located at:

  /Users/lewis/.juliaup/bin

This path will then be added to your PATH environment variable by
modifying the profile files located at:

  /Users/lewis/.zshrc

Julia will look for a new version of Juliaup itself every 1440 minutes when you start julia.

✔ Do you want to install with these custom configuration choices? · Proceed with installation
Now installing Juliaup
Checking for new Julia versions
Installing Julia 1.11.5+0.aarch64.apple.darwin14
Checking standard library notarization............................................................done.
Configured the default Julia version to be 'release'.
Julia was successfully installed on your system.

Depending on which shell you are using, run one of the following
commands to reload the PATH environment variable:

  . /Users/lewis/.zshrc

This is what I call a technical install and you essentially made the same point that curling to a shell is not a good way.

I would also recommend a symlink directly to the currently chosen Julia version’s binary.

This also seems a bit dodgy:

# >>> juliaup initialize >>>

# !! Contents within this block are managed by juliaup !!

path=('/Users/lewis/.juliaup/bin' $path)
export PATH

# <<< juliaup initialize <<<

After success, none of these options to juliaup are particularly friendly:

juliaup
The Julia Version Manager

Usage: juliaup <COMMAND>

Commands:
  default      Set the default Julia version
  add          Add a specific Julia version or channel to your system. Access via `julia +{channel}` e.g. `julia +1.6`
  link         Link an existing Julia binary to a custom channel name
  list         List all available channels
  override     Manage directory overrides
  update       Update all or a specific channel to the latest Julia version
  remove       Remove a Julia version from your system
  status       Show all installed Julia versions
  gc           Garbage collect uninstalled Julia versions
  config       Juliaup configuration
  self         Manage this juliaup installation
  completions  Generate tab-completion scripts for your shell
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

To launch a specific Julia version, use `julia +{channel}` e.g. `julia +1.6`.
Entering just `julia` uses the default channel set via `juliaup default`.

The above works because juliaup puts the symlink in its own directory where it has appropriate permissions and doesn’t mess with symlinks in other places and that’s a good thing. But, it seems to rely on putting juliaup at the front of the path list:

echo $PATH
/Users/lewis/.juliaup/bin:/Users/lewis/.local/bin:/Users/lewis/nim-2.0.8/bin:/Users/lewis/.nimble/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/TeX/texbin

This is the observed breakage for the LSP server in Sublime Text AFTER explicitly setting the path to the Julia binary reported by which julia:

The julia server has crashed 5 times in the last 180 seconds.

You can try to Restart it or you can choose Cancel to disable it for this window for the duration of the current session. Re-enable by running "LSP: Enable Language Server In Project" from the Command Palette.

This is the funky path problem by going through .juliaup and running julialauncher instead of julia. So, the julia binary is shrouded. What does julialuancher do? it should just contain the target of the chosen julia. This seems like an anti-pattern. I will try to solve with explicit symlink in /usr/local/bin to the actual julia binary, but this doesn’t work. Of course, this means I won’t be able to switch to a different julia binary. For my uses, I don’t need that capability. I may have had a permissions problem with my symlink, which I have fixed. Probably could have solved this with juliaup.

This output seems to munge the notions of channel and version. Currently, channel I thought referred to things like (incomplete list): nightly, release, rc. But, earlier on the release process hadn’t been as disciplined as it is now so there were just several builds of a version.

I’ve been super polite and respectful so far but I can’t see how any of this meets anyone’s definition of “friendly”. Sure, we will have different opinions of that, but c’mon.

Finally, it shows every chip implementation. On a given machine we have only one chip. Perhaps with a robust VM, you could cross-run other architectures but is that the intention. Wouldn’t it make sense to limit the list to the architecture of the current machine. Julia itself has a function to return that–surely Rust might provide a library function for that (or one could be written).

Now the juliaup status and juliaup update commands seem plenty friendly enough.

I still can get behind “one way, this way” for installation and management. But, there is a long way to go get to friendly. It looks like a lot of work, but if this mostly one-time investment can get Julia to a new place and save lots of future, fragmented work then by all means go for it.

2 Likes
28

Currently juliaup doesn’t play nice with symlinks since it doesn’t normalize it’s own path correctly. I’ve encountered similar issues to you in the past where I installed juliaup in a custom location. Symlinked to ~/.local/bin then decided to uninstall juliaup and endedup nuking my whole ~/.local folder. Why? because It thought that since the “executable path” was in ~/.local/bin then that must have also been the installation folder.

There have been some efforts (see: canonicalize the current_exe result when self updating, so that it works with symlinking. by ghyatzo · Pull Request #886 · JuliaLang/juliaup · GitHub, and later Centralize where paths are discovered by davidanthoff · Pull Request #915 · JuliaLang/juliaup · GitHub) but which have been gathering dust for quite a while now…

One of my major pain points with juliaup is the relative difficulty in contributing to it :frowning:

29

What should a GUI have at least, to provide enough features in your opinion?

30

From my point of view:

  1. select installation path (only after you choose ‘custom installation path’)
  2. install for all users (checkbox)
  3. select Julia version with the option to select more than one and select one as default
  4. install in start menu
  5. install starter icon
3 Likes
31

These all seem doable. I think the way to do this is to have a GUI installer that walks the user through installing juliaup with these options.

2 Likes
32

There are a lot of topics in this thread and in the end it’s unclear what the feature/change request is. I don’t want to re-open the “can of worms”, but I have a few opinions and want to contrast juliaup with other software managers, which no one has done yet.

TL;DR: I don’t think the Julia team needs to make any changes to the Julia install experience for typical users; it’s comparable to other software languages and version managers.

Windows DX

I would consider installing something in a non-standard directory an advanced user task. I think the Windows Store experience is perfectly fine and easier and more modern than an old-school Windows install wizard. So, IMO the Julia team shouldn’t be spending effort on that without good reason. In other words, my hunch is the kind of user that wants to install Julia in another location is more technical and able to follow an advanced install guide where they’d need to work on the command line, etc.

Maybe there’s an argument for it if Windows Store is being blocked by your organization, but if your Windows administrators are that strict, then they’re probably also blocking install wizards. Unfortunately, if you’re a software developer working in Windows, you basically need admin rights for half the software you need to install. We have this elevate user to admin privileges tool where I work.

EDIT: this thread is discussing a Windows GUI installer - A graphical Julia installer for Windows
There are organizations blocking the Windows Store but not installers.

Curl install, other software

StefanKarpinski

The installation process on other platforms is pretty simple but piping commands from curl into a shell is a little sketchy

I agree, but Julia is not a a standout here. These software version managers all install with curl and pipe to sh/bash.

Some software is even more sketchy and requires you to change powershell execution policy (e.g. uv). So if there’s a security concern here, it’s a conversation the entire software industry needs to have.

Software version managers are standard at this point and just another concept developers need to learn. In NodeJS world there’s fnm, which you can install from Windows store, or the older Linux/MacOS alternative - nvm.

Yes, they all have slightly different features and command arguments, which is confusing, but I don’t think juliaup is worse than others. As long as these tools have good default behavior and install latest stable version by default, then new users shouldn’t need to understand their details immediately or even look at juliaup help.

juliaup install experience

On juliaup installation itself. Yes, the wall of text is a bit overwhelming, but it is nice that juliaup is transparent about what it’s doing. You just need to skim the text and hit enter. Some text could be re-written to be less wordy, but overall I think it’s good and friendly.

You could argue this text should be minimalist and only output the install directories and confirmation question without all the explanatory sentences, but that’s a design choice people could argue about endlessly. In the end, you only need to read this text the few times you install juliaup, so unless this is scaring new users away, I’m not sure what could be improved here aside from minor text edits.

3 Likes
33

I disagree with this. I am a researcher who works on a 3–6 computational projects simultaneously, running on at least 5 machines of varying power, from my puny laptop to HPC servers. Each project is a package, with the manifest committed to the repo, so that we get the exact same environment. Upgrades, including Julia, require careful testing; I run into bugs occasionally even with Julia and then it is important to be able to backtrack.

On each machine, I have ssh access and manage the Julia installations using juliaup in a home directory. I find it extremely easy and flexible, and very convenient to interface with in scripts, eg when I am running multiple calculations on the same machine some have julia +1.10 in the shell scripts, while others julia +1.11.4 or julia +1.11.5. juliaup completes my dream for a reproducible environment.

19 Likes
34

I see that you’re installing on MacOS, which I don’t have experience with, but the Linux install options closely mirror the conda/Anaconda experience (in a good way). So I’ll just use that as a point of comparison.

I get that there is a lot of text, but I just did a fresh install of Miniconda (for a different reason), and the experience was pretty much exactly the same as juliaup. And typing conda in the terminal brings up docs that look exactly the same as juliaup. The only difference is conda manages packages, and juliaup manages language versions. So, I don’t really see a way around the text, unless the goal is to simply not explain anything.

The order of the path only matters if binaries/libraries/links exist in multiple places. Many systems have Python libraries shipped with the OS, but those are typically older and not used for programming. Systems will search the path for the first instance of a command, so Miniconda adding itself to the beginning of my PATH is actually a good thing. And the output of echo %PATH that you pasted means you should not symlink julia/juliaup anywhere else. Adding additional locations would either just cause problems or not do anything at all.

I think this is correct. I just did juliaup add nightly and it installed the latest version. I think the wording is just such because “version” doesn’t really make sense for things like nightly releases. So “nightly channel” compared to “version 1.11.5”

Out of curiosity, where do you see this? When I do juliaup status I only see the two versions/releases I explicitly added.

Is it though? Typically (ideally?) changing the directory is very simply typing the path the user wants, and that’s it. I changed the default path of my new conda install to ~/.local/miniconda3 because I like to keep my manual installs in the same place. And its’ pretty much set it and forget it after that point. And regardless of the location, I still need to point VS Code to the environment I want, so that doesn’t change.

I’ve tried to put on my '“absolute beginner’s hat” here, which may or may not have worked, and it seems that a lot of talk of the install being “technical” is just because its done via command line and show the text.

1 Like
35

While Juliaup is brilliant for switching between many cutting-edge and legacy versions, it would be nice if one eventually could also simply recommend new users to just go with the Julia version available from their favourite open-source package manager (apt, brew, etc.), to make their Julia install a bit less special in their environment. What are the reasons for this still not being the case? Are more comprehensive self-tests needed to help e.g. Linux distribution packagers to build Julia more reliably?

(One area where a juliaup installation is quite a bit of a complication is when you try to embed Julia into an application written in another language. You end up with somewhat unpredictable and frequently changing paths for the location of the libjulia library in the Makefile of the other-language application.)

36

Also, I would say that any version of julia that apt (or similar) would serve up would lag behind by quite a bit.

Maybe, some day, juliaup could be installed via apt (or similar) since it is simply a way to download prebuilt julia versions?

1 Like
37

Distros have gotten a bit more lenient about this in recent years, but the challenge has historically been that Julia uses very specific versions of multiple linked libraries. So specific, in fact, that Julia often carries patches that have been submitted upstream but aren’t generally available yet. But Linux distributions really dislike having programs “vendor” copies of libraries. They want to treat libraries like universal building blocks. And so they try to link Julia to whatever system LLVM and libm and libuv and PCRE etc that are already available in the package manager. LLVM is the most notorious offender here, and the preponderance of LLVM-based programming languages has helped socialize the understanding that — while it’s technically a library — it’s really a tightly-coupled internal bit of the language itself.

So I don’t think that’s really a blocker anymore; it’s mostly a problem of having someone do the work I think.

14 Likes
38

It depends on how this is done, if you host your own .deb repository like Signal you can have updates minutes after any new release (you can do something similar with Microsoft Edge). I have actually done most of this for Julia at https://taarn.srht.site/juliafromtar/README.html

39

Tamas: I did say it was “less (not no) benefit”. You are a very expert Julia user. Also, each of your 5 machines might have different Julia’s. But, your usage is a perfect example of someone testing/varying across Julia versions—what Juliaup is meant for.

To all: It really looks like the discussion has shifted from Juliaup in([:up, :down]) worm-can to practical ways to improve it. Perhaps the very focused suggestions might move to the package repo as Stefan suggested earlier. Just for clarity and to be part of the workflow for considering updates to juliaup and its “outside” manifestations, as the “inside” manifestations seem mostly good, if complicated (because the actual task IS complicated). That’s where the maintainers consider specific suggestions and indicate decisions about changes.

1 Like
40

I didn’t realize this was coming so soon – that is excellent and quite exciting! Is there a PR or issue opened up for this somewhere on juliaup?

(This could probably be its own topic but I do not have topic splitting powers.)

3 Likes
41

My Java background is showing LOL, so it seems to me juliaup does a good job of OOPS design for encapsulation / “inner mechanism hiding, but details still available if needed” like all good objects should.

So moving on About [quote=“StefanKarpinski, post:22, topic:128465”]
.. piping commands from curl into a shell is a little sketchy (it’s safe as long as the URL is HTTPS and you trust the server, but still) ..
[/quote]

With Ai now being used by sophisticated, sometimes state sponsored,
blackhat malware developers to such extent they have moved beyond
simple “typo squatting” (which IIRC thankfully @StefanKarpinski helped us guard against)
to :face_with_symbols_on_mouth: “Ai slop squatting” (AI hallucinations lead to a new cyber threat: Slopsquatting | CSO Online) I think we can see how blackhats could
be eager to exploit any security holes for such a fine powerful language
and community that Julia has become.

So, invoking the security principle of “reduce attack surface” or at least
double check / cross check and verify security precautions I am curious
if JuliaUp is still doing the suggested SHA256 checksum same as AWS ?
Here is why a SHA256 checksum after download is critical :
Security against malware - Howto verify download SHA256 checksum for eg Julia ISO file (Stop using DEB MD5 checksum) .. SHA-256 sums are now more frequently used by modern Linux distributions, as SHA-256 is more resistant to theoretical attacks. .. but the process is much more vulnerable - VERY BAD SECURITY RISK IF the attacker can replace the downloaded file (ISO) AND they can also replace the checksum file (or, GASP and software gods forbid and knock on wood, there is NO checksum !).

HTH and BTW I so quickly switch into :goblin: “devils advocate” or “what could go wrong” mode from another :innocent: whitehat penetration testing POV I hope everyone knows my intention is, not to be a critical pita, but to help the Julia community avoid headaches and malware attacks.
Attacks like the :face_with_symbols_on_mouth: “typo squatting” security issues some Python repositories seem to have UGH! which IIRC again thankfully @StefanKarpinski helped us guard against and completely avoid
.. so far.

1 Like