I think I have seen two different, unrelated projects that use ~/.julia/bin as a default directory for putting scripts and CLIs (and there might be others that I don’t know about):
I’m considering using such a convention in a project of mine as well, which led me to wonder:
Is this a (possibly emerging) standard practice?
Should we expect users to add this directory to their PATH environment variable?
And should we document this convention?
This might be a bit far fetched, but I tend to also relate this question to recurrent discussions about the best way to upgrade Julia (for example this one, following the release of v1.5.1): if it was common practice to add ~/.julia/bin to the PATH, then this would make it easier to have plug & play tooling that downloads a new version of julia and puts the relevant symlinks in there.
(PS: I’m using UNIX-like vocabulary here because that’s what I’m used to, but the idea should generalize to Windows as well)
I’ve also thought about documenting such a convention.
The next step is to make it more than a convention in the sense that the package manager knows about it and then we introduce the concept of a new type of package that is more like a standalone application (e.g. a CLI) in the package manager where it automatically puts it into .julia/bin. So you do something like pkg> add MyCLIApp which declares what “executables” it installs and the package manager handles everything for you and then you can just run it from the terminal.
Would require some design work but it might expand the scope of how Julia is used. It would be a bit similar to the binary/library distinction the Rust package manager Cargo has (where we currently only really have their equivalent of libraries).
Do you think it would be feasible for Pkg to also handle sysimage creation / update? (Maybe in a later stage, but I’m thinking such a feature might need to be planned ahead)
Maybe, but we probably want something better than the current sysimages first (like an incremental sysimage or something). Right now, they take too long to generate and are too big to be convenient to use.
It would be pretty nice for each standalone app to have it’s own sysimage because then it only needs to load e.g. the standard libararies it depends on (and not load OpenBLAS every time you call the julia ls command). One problem is that the sysimage is tied to the julia version so when you update Julia you would also need to rebuild all sysimages etc. This is a bit similar to that you need to re-precompile your packages when you update julia.
You can install mpiexecjl with MPI.install_mpiexecjl(). The default destination directory is joinpath(DEPOT_PATH[1], "bin") , which usually translates to ~/.julia/bin , but check the value on your system.
Please do not hardwire this to /home/$USERNAME/.julia/bin
Use an environment variable, which means it is possible to use the Modules environment on HPC cyctems etc, to change where these are stored, or to have multiple Julia vversions co-existing on a system
I work on HPC clusters. One curious thing - on a Linux system you must have a $HOME directory mounted. Lots of software squirrels away configurations in . files and . directories
Also the wheels fall off on login sessions of your .bashrc etc. is not available.
However consider:
there are setups where your $HOME directory is different from what you see as an external campus user
$HOME directories can be on filesystems with small capacities and low performance.
As I say you should have a $HOME directory. But let’s say you want to launch something on thousands of compute nodes. You want that code of a fast parallel filesystem - not some NFS server.
I have seen one setup where the $HOME directory is on an external NFS NAS campus server - so you have the same view of your $HOME directory wherever you are. Which is just fine.
But you don;t want tp be pulling executables from that NAS if you can avoid it.
ps. for those of you curious about my statement about needing a $HOME directory I found this out the hard way on a system which failed to mount /home…
You may not need a ‘real’ copy of your $HOME but you need something there!
That’s exactly what I was thinking about. It should not be too difficult to arrange for a sysimage to be created at Pkg.build time. However, I don’t know how to exert any control over the julia version used.
An option is to detect everything at run-time, use a suitable sysimage if one is found, and create a new one if needed (possibly launching a system image compilation in the background, while running the “program” without system image the first time). This could be handled by proxy scripts in ~/.julia/bin and avoids relying on new features in Julia or Pkg. (And the common logic could be standardized in a package, and re-used by every project needing it)
This makes me think that re-precompilation is probably already triggered by such run-time checks. Do you know whether there is an exposed API to do this? I imagine something like computing a slug based on the julia version and the package version (or the Manifest contents) or anything that would be relevant to determine whether a .ji file is stale. I’m thinking the same criteria would apply for sysimages as well, but maybe that assumption is wrong…