# Installation on managed cluster

**URL:** <https://discourse.julialang.org/t/installation-on-managed-cluster/100145>\
**Category:** General Usage\
**Tags:** cluster, installation\
**Created:** [June 10, 2023, 5:12pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145 "2023-06-10T17:12:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 10, 2023, 5:12pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/1 "2023-06-10T17:12:54Z")

</div>

I install scientific software on a big supercomputer cluster, and we have occasional requests for Julia. So I thought I’d make an installation that contains a bunch of popular packages such as HDF5 and MPI.

- Installing from source was trivial. Kudos!
- Setting `JULIA_DEPOT_PATH` I was able to get some packages to appear in a system location, rather than my home directory.
- However, telling a user about my executable and value of depot path makes them try to write things like log files in that system location.

So, as a complete non-user of Julia, meaning that you have to spell things out for me, I could use some help in how to set up a central installation.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 10, 2023, 8:05pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/2 "2023-06-10T20:05:07Z")

</div>

I think that users won’t benefit a lot from having local package installations. Julia will compile the code for each user anyway and store it in the user’s home directory (at the .julia dir).

It could be useful to have a local mirror of the general registry, but that’s only worthwhile if the user access to external sites is slow or cumbersome for any reason.

(even having a centralized installation of Julia itself is probably not very useful. Just downloading the binaries or managing Julia with juliaup is simple enough so that users shouldn’t have many issues. I’m guessing the cluster has some Linux flavor installed)

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [June 10, 2023, 8:10pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/3 "2023-06-10T20:10:08Z")

</div>

I’m going to agree with @lmiq here. I work with Julia on a cluster. Our cluster uses EasyBuild to configure software installations. I have had no issues using EasyBuild to install Julia in my home directory – it configures everything correctly. Once I load it with `module load Julia/specific-version-name` it works just as it does on my local computer. The bonus of this is that I can upgrade as soon as I want (I installed 1.9.1 this morning).

---

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 10, 2023, 9:56pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/4 "2023-06-10T21:56:22Z")

</div>

I see your point. But.

“Just downloading the binaries”

Right. This whole issue started when a user observed that the downloaded binary was factors slower than expected. My install from source was much faster, presumaby because the compiler could find a better instruction set and do other observations and whatnot.

Never use the word “just” around me. Please.

---

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 10, 2023, 9:59pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/5 "2023-06-10T21:59:42Z")

</div>

> [@tbeason](#):
>
> it configures everything correctly

Does that include the MPI module finding the right network? We ban, eh, sternly chide, users for running parallel jobs through the generic ethernet network rather than our expensive high speed network. If you tell me that “add MPI” finds the right network, then I’m happy.

(So that was going to be my next question: how to tell the MPI module to find the right network.)

---

<div class="post-metadata">

**Author:** ![Ralph\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ralph_smith/32/10344_2.png) [@Ralph\_Smith](https://discourse.julialang.org/u/Ralph_Smith)\
**Post date:** [June 10, 2023, 10:34pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/6 "2023-06-10T22:34:26Z")

</div>

The docs for the Julia MPI interface have instructions for setting things up in cases like yours [here](https://juliaparallel.org/MPI.jl/stable/configuration) There is a section specific to the cluster case.

---

<div class="post-metadata">

**Author:** ![tbeason](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tbeason/32/15898_2.png) [@tbeason](https://discourse.julialang.org/u/tbeason)\
**Post date:** [June 10, 2023, 10:55pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/7 "2023-06-10T22:55:07Z")

</div>

I guess I was too loose with my language when I said everything haha. I don’t use things like MPI (I just use a bigger node!).

---

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 10, 2023, 11:00pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/8 "2023-06-10T23:00:50Z")

</div>

> [@Ralph\_Smith](#):
>
> There is a section specific to the cluster case.

Cool. But it doesn’t look like I can set that up for users, right? I’ll have to make very clear instructions for anyone wanting to install Julia.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [June 10, 2023, 11:23pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/9 "2023-06-10T23:23:28Z")

</div>

No, if you follow the instructions [further down on that page](https://juliaparallel.org/MPI.jl/stable/configuration/#Notes-to-HPC-cluster-administrators), your users shouldn’t need to do anything besides `add MPI` to access whatever system-specific MPI binary you’ve specified.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [June 11, 2023, 1:08am UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/10 "2023-06-11T01:08:03Z")

</div>

I recommend looking into how JLL packages work and how to override them.

[https://docs.binarybuilder.org/stable/jll/#Overriding-the-artifacts-in-JLL-packages](https://docs.binarybuilder.org/stable/jll/#Overriding-the-artifacts-in-JLL-packages)

Also see the docs on hoe deoots stack and code loading

[https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT\_PATH](https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT_PATH)

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [June 11, 2023, 8:44am UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/11 "2023-06-11T08:44:18Z")

</div>

Wow @VictorEijkhout has joined the Julia discourse! Victor, I have your book Introduction to High Performance Scientific Computing on my bookshelf.

Victors books are here: [http://theartofhpc.com/](http://theartofhpc.com/)

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 11, 2023, 3:27pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/12 "2023-06-11T15:27:23Z")

</div>

This here might be helpful: [GitHub - hlrs-tasc/julia-on-hpc-systems: Information on how to set up Julia on HPC systems](https://github.com/hlrs-tasc/julia-on-hpc-systems)

Note that it is a bit outdated though.

I can’t share our setup right now because I’m on my phone. Feel free to ping me again if you haven’t heard from me later this week.

cc @sloede

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 11, 2023, 3:52pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/13 "2023-06-11T15:52:19Z")

</div>

> [@VictorEijkhout](#):
>
> Right. This whole issue started when a user observed that the downloaded binary was factors slower than expected. My install from source was much faster, presumaby because the compiler could find a better instruction set and do other observations and whatnot.

What was slow specifically? The Julia distribution is mainly a compiler + a runtime, unless you’re hitting a specific workflow where the runtime matters a lot (for example the garbage collector), the common knowledge is that what code the compiler generates shouldn’t depend much (if at all) on how the compiler itself was compiled, i.e. targeting the generic base architecture or the exact ISA of the current CPU.

---

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 11, 2023, 4:59pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/14 "2023-06-11T16:59:41Z")

</div>

> [@giordano](#):
>
> the common knowledge is that what code the compiler generates

You’re on the wrong level. I think the observation was that the compiler itself was slow. For instance in starting up the Julia executable to begin with.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 11, 2023, 5:08pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/15 "2023-06-11T17:08:58Z")

</div>

So what you observed to be slow wasn’t the code generated by the compiler, that’s why I asked _what_ was slow 🙂

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [June 12, 2023, 5:40am UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/16 "2023-06-12T05:40:49Z")

</div>

I should imagine that @VictorEijkhout knows this very well… often on HPC systems the users home directory is on a slw and space limited NFS filesystem. The working storage is on faster parallel storage with high capacity. Is this a factor here?

---

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 12, 2023, 1:59pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/17 "2023-06-12T13:59:51Z")

</div>

> [@johnh](#):
>
> Is this a factor here?

Combination of space considerations and that I like to provide optimized MPI, maybe HDF5, other packages where the context makes a difference.

---

<div class="post-metadata">

**Author:** ![sloede](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sloede/32/44787_2.png) [@sloede](https://discourse.julialang.org/u/sloede)\
**Post date:** [June 12, 2023, 3:03pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/18 "2023-06-12T15:03:32Z")

</div>

> [@johnh](#):
>
> often on HPC systems the users home directory is on a slw and space limited NFS filesystem. The working storage is on faster parallel storage with high capacity. Is this a factor here?

For all practical intents, in our experience the location of the Julia depot is usually _not_ a performance-critical factor before you scale to \>1000 MPI ranks (at least since Julia v1.7).

@VictorEijkhout You should be able to use centrally managed Preferences.jl and/or environment variables to configure the correct settings for MPI.jl and HDF5.jl. For this, you need to put a centrally managed `LocalPreferences.jl` file in the users’ `JULIA_LOAD_PATH` (e.g., by setting the `JULIA_LOAD_PATH` appropriately in a module file). In case of environment variables, you can just set them directly inside, e.g., the Julia module. The contents of the preferences file or the relevant environment variables can be found in the MPI.jl/HDF5.jl docs (let me know if you have trouble finding the right place).

For our purposes, we have found that maintaining centrally managed package installations in a shared depot is not worth the hassle. Even on systems where users cannot access the internet directly, there are ways around that (see, e.g., [this topic](https://discourse.julialang.org/t/installing-packages-via-an-ssh-socks-proxy-on-a-compute-cluster/71735)).

---

<div class="post-metadata">

**Author:** ![VictorEijkhout](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/victoreijkhout/32/50666_2.png) [@VictorEijkhout](https://discourse.julialang.org/u/VictorEijkhout)\
**Post date:** [June 12, 2023, 4:06pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/19 "2023-06-12T16:06:04Z")

</div>

> [@stillyslalom](#):
>
> if you follow the instructions

Thanks. That was fairly clear. Eh,

```julia
[staff julia:132] cat ${juliaroot}/depot/environments/v1.10/LocalPreferences.toml
[MPIPreferences]
_format = "1.0"
abi = "MPICH"
binary = "system"
libmpi = "libmpi"
mpiexec = "mpiexec"

```

I had expected to see a path to the `libmpi` and a trailing `.so`. Should I edit that? Is there a way to run `ldd` on a Julia executable to see where it gets its MPI? (Yes, I know. Mutatis mutandis.)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [June 12, 2023, 4:38pm UTC](https://discourse.julialang.org/t/installation-on-managed-cluster/100145/20 "2023-06-12T16:38:48Z")

</div>

> [@VictorEijkhout](#):
>
> I had expected to see a path to the `libmpi` and a trailing `.so`.

The value of `libmpi` is whatever you can pass to `dlopen` to find the library. The basename without extension is fine for the dynamic loader, and it’s generally preferable because it’s platform-independent. Of course this doesn’t matter much when you’re doing system-specific configuration.

> [@VictorEijkhout](#):
>
> Is there a way to run `ldd` on a Julia executable to see where it gets its MPI?

Not really, Julia dynamically loads shared libraries like plugins with `dlopen`. After you have installed the `MPI.jl` package, you can see the full path of libmpi with

```julia
using MPI
MPI.API.libmpi

```

But if you want to be extra sure of what library to use, you can also use the absolute path of your libmpi as `libmpi` value, that also works for `dlopen`.

[Next page](https://discourse.julialang.org/t/installation-on-managed-cluster/100145.md?page=2)