# Running CUDA.jl with JULIA\_DEPOT\_PATH read-only

**URL:** <https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833>\
**Category:** GPU\
**Tags:** question\
**Created:** [June 18, 2024, 11:15pm UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833 "2024-06-18T23:15:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![erin-atkinson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erin-atkinson/32/50713_2.png) [@erin-atkinson](https://discourse.julialang.org/u/erin-atkinson)\
**Post date:** [June 18, 2024, 11:15pm UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/1 "2024-06-18T23:15:09Z")

</div>

I’m updating from Julia 1.8 to 1.10 for the simulations I’m running. Previously everything worked fine, but now I have an issue. I can install CUDA.jl just fine and use it on the HPC login nodes, but on compute nodes the home filesystem is read-only, and CUDA always tries to pre-compile, even if it’s already been pre-compiled on a login node.  
If I put the Julia DEPOT\_PATH into the scratch filesystem, it works just fine because it can write to it.  
Why has this started happening, when it wasn’t an issue before (CUDA version 4.4 → 5.3)? Can I force it to not pre-compile?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 19, 2024, 7:12am UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/2 "2024-06-19T07:12:23Z")

</div>

I don’t think it’s ideal to use a read-only primary depot, as packages like GPUCompiler.jl expects to be able to use read-write scratchspaces, which are also stored in the depot. Can you not use a layered depot, with the primary entry still being read-write?

Regardless of that, the package is not expected to recompile again. Try running with `JULIA_DEBUG=loading` to inspect why this happens.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 19, 2024, 7:42am UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/3 "2024-06-19T07:42:16Z")

</div>

Typically, you should put your Julia depot on the parallel filesystem (PFS). Often times, that’s “scratch”.

---

<div class="post-metadata">

**Author:** ![erin-atkinson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erin-atkinson/32/50713_2.png) [@erin-atkinson](https://discourse.julialang.org/u/erin-atkinson)\
**Post date:** [June 19, 2024, 6:18pm UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/4 "2024-06-19T18:18:09Z")

</div>

Ah, I did expect it was an unusual use case, it’s okay if this is just not supported, I just don’t want to gunk up this filesystem but I suppose I must…  
What do you mean by a layered depot?

Anyway here are the loading debug messages, seems to stem from an edited libcuda:

```julia
julia> using CUDA
┌ Debug: Rejecting stale cache file /home/n/ngrisoua/atkin163/.julia-mist/compiled/v1.10/CUDA_Runtime_jll/Hs50y_SwiJs.ji (mtime 1.717012621771995e9) because file /lib64/libcuda.so.1 (mtime 1.7170137474668772e9) has changed
└ @ Base loading.jl:3143
┌ Debug: Ignoring cache file /home/n/ngrisoua/atkin163/.julia-mist/compiled/v1.10/CUDA_Runtime_jll/Hs50y_3AysM.ji for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2] (fafbfcfd-01e2-611a-0006-41bd32282031) since it is does not provide desired build_id (fafbfcfd-f246-903e-0006-43c02359b684)
└ @ Base loading.jl:3058
┌ Debug: Rejecting cache file /home/n/ngrisoua/atkin163/.julia-mist/compiled/v1.10/CUDA/oWw5k_SwiJs.ji because required dependency CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2] with build ID fafbfcfd-f246-903e-0006-43c02359b684 is missing from the cache.
└ @ Base loading.jl:1560
┌ Debug: Rejecting cache file /home/n/ngrisoua/atkin163/.julia-mist/compiled/v1.10/CUDA/oWw5k_3AysM.ji because it is for file /home/n/ngrisoua/atkin163/scratch/.julia-mist/packages/CUDA/B2Z5u/src/CUDA.jl not file /home/n/ngrisoua/atkin163/.julia-mist/packages/CUDA/B2Z5u/src/CUDA.jl
└ @ Base loading.jl:3113
ERROR: IOError: open("/gpfs/fs1/home/n/ngrisoua/atkin163/.julia-mist/logs/manifest_usage.toml.pid", 194, 292): read-only file system (EROFS)
...

```

For the sake of completion here is CUDA.versioninfo()

```julia
CUDA runtime 11.8, artifact installation
CUDA driver 11.7
NVIDIA driver 515.105.1

CUDA libraries:
- CUBLAS: 11.11.3
- CURAND: 10.3.0
- CUFFT: 10.9.0
- CUSOLVER: 11.4.1
- CUSPARSE: 11.7.5
- CUPTI: 18.0.0
- NVML: 11.0.0+515.105.1

Julia packages:
- CUDA: 5.3.5
- CUDA_Driver_jll: 0.8.1+0
- CUDA_Runtime_jll: 0.12.1+0

Toolchain:
- Julia: 1.10.4
- LLVM: 15.0.7

4 devices:
  0: Tesla V100-SXM2-32GB (sm_70, 25.215 GiB / 32.000 GiB available)
  1: Tesla V100-SXM2-32GB (sm_70, 26.889 GiB / 32.000 GiB available)
  2: Tesla V100-SXM2-32GB (sm_70, 26.889 GiB / 32.000 GiB available)
  3: Tesla V100-SXM2-32GB (sm_70, 27.397 GiB / 32.000 GiB available)

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 19, 2024, 7:23pm UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/5 "2024-06-19T19:23:21Z")

</div>

I suspected as much. In this case, it’s kind-of needed to recompile, as the CUDA driver (`libcuda`) determines what toolkit you can use. So if the driver changes, users would want CUDA.jl to recompile so that we’re using a compatible CUDA toolkit.

To prevent this, you would need to make sure the same driver is being used on the compute nodes, or you would need to manually specify the toolkit to use (see here for how that logic works, [Yggdrasil/C/CUDA/CUDA\_Runtime/platform\_augmentation.jl at 7d67bcabd520057dffa52e9f160afc8fef24d720 · JuliaPackaging/Yggdrasil · GitHub](https://github.com/JuliaPackaging/Yggdrasil/blob/7d67bcabd520057dffa52e9f160afc8fef24d720/C/CUDA/CUDA_Runtime/platform_augmentation.jl#L62-L82)). This can be done using a preference, which you can set by calling `CUDA.set_runtime_version!`.

---

<div class="post-metadata">

**Author:** ![erin-atkinson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erin-atkinson/32/50713_2.png) [@erin-atkinson](https://discourse.julialang.org/u/erin-atkinson)\
**Post date:** [June 19, 2024, 9:00pm UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/6 "2024-06-19T21:00:09Z")

</div>

I did that (`CUDA.set_runtime_version!(v"11.8.0"; local_toolkit=true)`), and it does manage to get further, but then this:  
It looks like in the cuda library folder there is no nvperf\_host? Is this an issue I need to bring up with the support?

```julia
┌ Debug: Looking for library nvperf_host, no specific version, in /scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0 or /scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/extras/compute-sanitizer or /scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/compute-sanitizer
│ all_names =
│ 1-element Vector{String}:
│ "libnvperf_host.so"
│ all_locations =
│ 15-element Vector{String}:
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0"
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/lib"
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/lib64"
│ ⋮
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/compute-sanitizer/libx64"
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/compute-sanitizer/targets/ppc64le-linux/lib"
└ @ CUDA_Runtime_Discovery ~/.julia-mist/packages/CUDA_Runtime_Discovery/ZGnCZ/src/CUDA_Runtime_Discovery.jl:136
┌ Debug: Did not find nvperf_host
└ @ CUDA_Runtime_Discovery ~/.julia-mist/packages/CUDA_Runtime_Discovery/ZGnCZ/src/CUDA_Runtime_Discovery.jl:139
┌ Debug: Looking for library nvperf_host, version 9.0.0 or 9.1.0 or 9.2.0 or 10.0.0 or 10.1.0 or 10.2.0 or 11.0.0 or 11.1.0 or 11.2.0 or 11.3.0 or 11.4.0 or 11.5.0 or 11.6.0 or 11.7.0 or 11.8.0 or 12.0.0 or 12.1.0 or 12.2.0 or 12.3.0 or 12.4.0 or 12.5.0 or 12.6.0 or 12.7.0 or 12.8.0 or 12.9.0 or 12.10.0 or 13.1.0 or 13.2.0 or 13.3.0 or 13.4.0 or 13.5.0 or 13.6.0 or 13.7.0 or 13.8.0 or 13.9.0 or 13.10.0 or 14.1.0 or 14.2.0 or 14.3.0 or 14.4.0 or 14.5.0 or 14.6.0 or 14.7.0 or 14.8.0 or 14.9.0 or 14.10.0, in /scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0 or /scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/extras/compute-sanitizer or /scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/compute-sanitizer
│ all_names =
│ 139-element Vector{String}:
│ "libnvperf_host.so"
│ "libnvperf_host.so.9.0.0"
│ "libnvperf_host.so.9.0"
│ ⋮
│ "libnvperf_host.so.14.10"
│ "libnvperf_host.so.14"
│ all_locations =
│ 15-element Vector{String}:
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0"
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/lib"
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/lib64"
│ ⋮
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/compute-sanitizer/libx64"
│ "/scinet/mist/rhel8/software/2021a/opt/base/cuda/11.8.0/compute-sanitizer/targets/ppc64le-linux/lib"
└ @ CUDA_Runtime_Discovery ~/.julia-mist/packages/CUDA_Runtime_Discovery/ZGnCZ/src/CUDA_Runtime_Discovery.jl:136
┌ Debug: Did not find nvperf_host
└ @ CUDA_Runtime_Discovery ~/.julia-mist/packages/CUDA_Runtime_Discovery/ZGnCZ/src/CUDA_Runtime_Discovery.jl:139
┌ Debug: Could not discover CUDA toolkit
│ exception =
│ Could not find library 'nvperf_host' in your local CUDA installation.

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 20, 2024, 10:59am UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/7 "2024-06-20T10:59:51Z")

</div>

This is fixed in the latest CUDA\_Runtime\_Discovery.jl; but you’re using and old version (v0.2.4). Upgrading to CUDA v5.4 should fix this.

---

<div class="post-metadata">

**Author:** ![erin-atkinson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erin-atkinson/32/50713_2.png) [@erin-atkinson](https://discourse.julialang.org/u/erin-atkinson)\
**Post date:** [June 21, 2024, 1:10am UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/8 "2024-06-21T01:10:51Z")

</div>

Ah, thank you. Unfortunately this cluster uses Power9 CPUs…I bit the bullet and just put the depot in scratch.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [June 21, 2024, 7:55am UTC](https://discourse.julialang.org/t/running-cuda-jl-with-julia-depot-path-read-only/115833/9 "2024-06-21T07:55:22Z")

</div>

If you need specific features, such as compatibility with the latest CUDA\_Runtime\_Discovery\_jll, feel free to create a backport PR. There is already a release branch: [GitHub - JuliaGPU/CUDA.jl at release-5.3](https://github.com/JuliaGPU/CUDA.jl/tree/release-5.3)