# ~/.julia/bin as a standard location for scripts?

**URL:** <https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993>\
**Category:** Tooling\
**Created:** [September 3, 2020, 8:17am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993 "2020-09-03T08:17:22Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 3, 2020, 8:17am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/1 "2020-09-03T08:17:23Z")

</div>

I think I have seen two different, unrelated projects that use `~/.julia/bin` as a default directory for putting scripts and CLIs (and there might be others that I don’t know about):

- [`jlpkg`](https://github.com/fredrikekre/jlpkg)
- [`comonicon`](https://github.com/Roger-luo/Comonicon.jl) and tools built upon it (e.g. [IonCLI](https://github.com/Roger-luo/IonCLI.jl))

I’m considering using such a convention in a project of mine as well, which led me to wonder:

- Is this a (possibly emerging) standard practice?
- Should we expect users to add this directory to their `PATH` environment variable?
- And should we document this convention?

  

This might be a bit far fetched, but I tend to also relate this question to recurrent discussions about the best way to upgrade Julia (for example [this one](https://discourse.julialang.org/t/is-there-a-simple-julia-upgrade-command/45691), following the release of v1.5.1): if it was common practice to add `~/.julia/bin` to the `PATH`, then this would make it easier to have plug & play tooling that downloads a new version of julia and puts the relevant symlinks in there.

  

* * *

_(PS: I’m using UNIX-like vocabulary here because that’s what I’m used to, but the idea should generalize to Windows as well)_

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 3, 2020, 8:28am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/2 "2020-09-03T08:28:57Z")

</div>

I’ve also thought about documenting such a convention.

The next step is to make it more than a convention in the sense that the package manager knows about it and then we introduce the concept of a new type of package that is more like a standalone application (e.g. a CLI) in the package manager where it automatically puts it into `.julia/bin`. So you do something like `pkg> add MyCLIApp` which declares what “executables” it installs and the package manager handles everything for you and then you can just run it from the terminal.

Would require some design work but it might expand the scope of how Julia is used. It would be a bit similar to the binary/library distinction the Rust package manager Cargo has (where we currently only really have their equivalent of libraries).

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 3, 2020, 8:30am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/3 "2020-09-03T08:30:39Z")

</div>

> [@kristoffer.carlsson](#):
>
> The next step is to make it more than a convention in the sense that the package manager knows about it and then we introduce the concept of a new type of package that is more like a standalone application

That would be awesome!

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [September 3, 2020, 8:33am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/4 "2020-09-03T08:33:44Z")

</div>

Also `MPI.jl` installs a script to `~/.julia/bin`: [Configuration · MPI.jl](https://juliaparallel.github.io/MPI.jl/stable/configuration/#Installation). But I guess we all followed the example set by `jlpkg`

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 3, 2020, 8:37am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/5 "2020-09-03T08:37:11Z")

</div>

> [@kristoffer.carlsson](#):
>
> the package manager handles everything for you and then you can just run it from the terminal.
> 
> Would require some design work but it might expand the scope of how Julia is used.

Do you think it would be feasible for `Pkg` to also handle sysimage creation / update? (Maybe in a later stage, but I’m thinking such a feature might need to be planned ahead)

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 3, 2020, 8:43am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/6 "2020-09-03T08:43:12Z")

</div>

> [@ffevotte](#):
>
> Do you think it would be feasible for `Pkg` to also handle sysimage creation / update?

Maybe, but we probably want something better than the current sysimages first (like an incremental sysimage or something). Right now, they take too long to generate and are too big to be convenient to use.

It would be pretty nice for each standalone app to have it’s own sysimage because then it only needs to load e.g. the standard libararies it depends on (and not load OpenBLAS every time you call the julia `ls` command). One problem is that the sysimage is tied to the julia version so when you update Julia you would also need to rebuild all sysimages etc. This is a bit similar to that you need to re-precompile your packages when you update julia.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 3, 2020, 9:45am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/7 "2020-09-03T09:45:49Z")

</div>

Note what MPI.lj says

You can install `mpiexecjl` with [`MPI.install_mpiexecjl()`](https://juliaparallel.github.io/MPI.jl/stable/environment/#MPI.install_mpiexecjl). The default destination directory is `joinpath(DEPOT_PATH[1], "bin")` , which usually translates to `~/.julia/bin` , but check the value on your system.

Please do not hardwire this to /home/$USERNAME/.julia/bin  
Use an environment variable, which means it is possible to use the Modules environment on HPC cyctems etc, to change where these are stored, or to have multiple Julia vversions co-existing on a system

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [September 3, 2020, 9:53am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/8 "2020-09-03T09:53:50Z")

</div>

Note that

1. the `DEPOT_PATH` Julia variable can be controlled by the [`JULIA_DEPOT_PATH`](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_DEPOT_PATH) environment variable

2. you can pass whatever path you want to [`MPI.install_mpiexecjl()`](https://juliaparallel.github.io/MPI.jl/stable/environment/#MPI.install_mpiexecjl) using the `destdir` keyword argument

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 3, 2020, 10:00am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/9 "2020-09-03T10:00:55Z")

</div>

Agreeing with you.

I work on HPC clusters. One curious thing - on a Linux system you must have a $HOME directory mounted. Lots of software squirrels away configurations in . files and . directories  
Also the wheels fall off on login sessions of your .bashrc etc. is not available.

However consider:  
there are setups where your $HOME directory is different from what you see as an external campus user  
$HOME directories can be on filesystems with small capacities and low performance.

As I say you should have a $HOME directory. But let’s say you want to launch something on thousands of compute nodes. You want that code of a fast parallel filesystem - not some NFS server.

I have seen one setup where the $HOME directory is on an external NFS NAS campus server - so you have the same view of your $HOME directory wherever you are. Which is just fine.  
But you don;t want tp be pulling executables from that NAS if you can avoid it.

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [September 3, 2020, 10:02am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/10 "2020-09-03T10:02:19Z")

</div>

ps. for those of you curious about my statement about needing a $HOME directory I found this out the hard way on a system which failed to mount /home…  
You may not need a ‘real’ copy of your $HOME but you need something there!

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 3, 2020, 10:22am UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/11 "2020-09-03T10:22:26Z")

</div>

> [@kristoffer.carlsson](#):
>
> One problem is that the sysimage is tied to the julia version so when you update Julia you would also need to rebuild all sysimages etc.

That’s exactly what I was thinking about. It should not be too difficult to arrange for a sysimage to be created at `Pkg.build` time. However, I don’t know how to exert any control over the julia version used.

An option is to detect everything at run-time, use a suitable sysimage if one is found, and create a new one if needed (possibly launching a system image compilation in the background, while running the “program” without system image the first time). This could be handled by proxy scripts in `~/.julia/bin` and avoids relying on new features in Julia or `Pkg`. (And the common logic could be standardized in a package, and re-used by every project needing it)

> [@kristoffer.carlsson](#):
>
> This is a bit similar to that you need to re-precompile your packages when you update julia.

This makes me think that re-precompilation is probably already triggered by such run-time checks. Do you know whether there is an exposed API to do this? I imagine something like computing a slug based on the julia version and the package version (or the Manifest contents) or anything that would be relevant to determine whether a .ji file is stale. I’m thinking the same criteria would apply for sysimages as well, but maybe that assumption is wrong…

---

<div class="post-metadata">

**Author:** ![ffevotte](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ffevotte/32/6587_2.png) [@ffevotte](https://discourse.julialang.org/u/ffevotte)\
**Post date:** [September 3, 2020, 12:00pm UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/12 "2020-09-03T12:00:42Z")

</div>

> [@giordano](#):
>
> you can pass whatever path you want to [`MPI.install_mpiexecjl()`](https://juliaparallel.github.io/MPI.jl/stable/environment/#MPI.install_mpiexecjl) using the `destdir` keyword argument

Incidentally, an API like

```julia
julia> MyTool.install(destdir="/path/to/custom/dir")

```

again as pioneered by `jlpkg`, is probably what we’d more generally want to standardize/encourage (unless/until dedicated support arrives in `Pkg`).

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 3, 2020, 12:01pm UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/13 "2020-09-03T12:01:02Z")

</div>

> [@ffevotte](#):
>
> Do you know whether there is an exposed API to do this? For example, computing a slug based on the julia version and the package version (or the Manifest contents) or anything that would be relevant to determine whether a .ji file is stale.

There isn’t really an exposed API but there is staleness checks and slug generations

> <https://github.com/JuliaLang/julia/blob/b5956a89d456af00278cca75ac1ef3fcfaa2d4a1/base/loading.jl#L1363-L1446>

and

> <https://github.com/JuliaLang/julia/blob/b5956a89d456af00278cca75ac1ef3fcfaa2d4a1/base/loading.jl#L144-L153>

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [September 3, 2020, 12:29pm UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/14 "2020-09-03T12:29:39Z")

</div>

Wrote some stuff here [https://github.com/JuliaLang/Pkg.jl/issues/1962#issuecomment-686447390](https://github.com/JuliaLang/Pkg.jl/issues/1962#issuecomment-686447390).

---

<div class="post-metadata">

**Author:** ![rapus95](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rapus95/32/3773_2.png) [@rapus95](https://discourse.julialang.org/u/rapus95)\
**Post date:** [September 10, 2020, 6:36pm UTC](https://discourse.julialang.org/t/julia-bin-as-a-standard-location-for-scripts/45993/15 "2020-09-10T18:36:55Z")

</div>

x-ref [https://github.com/JuliaLang/julia/issues/34552#issuecomment-579619366](https://github.com/JuliaLang/julia/issues/34552#issuecomment-579619366)