CUDA and NVTX fail to precompile on cluster

Specific Domains GPU
,
1

I am trying to use CUDA on my university cluster. If I try

‘’’ using CUDA ‘’’

I get the following error(s) (I have removed the Stacktraces):

‘’’
ERROR: LoadError: InitError: could not load library “/path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so”

/path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so: cannot open shared object file: No such file or directory

ERROR: LoadError: Failed to precompile NVTX [5da4648a-3479-48b8-97b9-01cb529c0a1f] to “/path-to-home/.julia/compiled/v1.10/NVTX/jl_QzKBuC”.

ERROR: The following 2 direct dependencies failed to precompile:

NVTX [5da4648a-3479-48b8-97b9-01cb529c0a1f]

Failed to precompile NVTX [5da4648a-3479-48b8-97b9-01cb529c0a1f] to “/path-to-home/.julia/compiled/v1.10/NVTX/jl_8DfuJi”.

ERROR: LoadError: InitError: could not load library “/path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so”

/path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so: cannot open shared object file: No such file or directory

Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to “path-to-home/.julia/compiled/v1.10/CUDA/jl_2MEhVu”.

ERROR: LoadError: InitError: could not load library “/path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so”

/path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so: cannot open shared object file: No such file or directory

ERROR: LoadError: Failed to precompile NVTX [5da4648a-3479-48b8-97b9-01cb529c0a1f] to “/path-to-home/.julia/compiled/v1.10/NVTX/jl_QzKBuC”.

‘’’
The julia version is 1.10.5

How can I fix this? Any help would be appreciated.

2

Does /path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so exist? Can you do ldd /path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1/lib/libnvToolsExt.so?

3

No, the file does not exist.

4

That’s obviously a problem. Did you see any error messages when adding the packages to the Julia environment, e.g. during recompilation?

5

I removed CUDA and tried adding it again, the error message is this:

Precompiling project...
  ✗ CUDA → SpecialFunctionsExt
  11 dependencies successfully precompiled in 182 seconds. 83 already precompiled. 2 skipped during auto due to previous errors.
6

I should add that some dependencies had the outdated version installed

  [621f4979] + AbstractFFTs v1.5.0
  [a9b6321e] + Atomix v0.1.0
  [ab4f0b2a] + BFloat16s v0.5.0
  [fa961155] + CEnum v0.5.0
  [052768ef] + CUDA v5.4.3
  [1af6417a] + CUDA_Runtime_Discovery v0.3.5
  [34da2185] + Compat v4.16.0
  [9a962f9c] + DataAPI v1.16.0
  [a93c6f00] + DataFrames v1.6.1
  [864edb3b] + DataStructures v0.18.20
  [e2d170a0] + DataValueInterfaces v1.0.0
  [e2ba6199] + ExprTools v0.1.10
  [0c68f7d7] + GPUArrays v10.3.1
⌅ [61eb1bfa] + GPUCompiler v0.26.7
  [842dd82b] + InlineStrings v1.4.2
  [41ab1584] + InvertedIndices v1.3.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [63c18a36] + KernelAbstractions v0.9.25
⌅ [929cbde3] + LLVM v8.1.0
  [8b046642] + LLVMLoopInfo v1.0.0
  [b964fa9f] + LaTeXStrings v1.3.1
  [e1d29d7a] + Missings v1.2.0
  [2dfb63ee] + PooledArrays v1.4.3
  [08abe8d2] + PrettyTables v2.3.2
  [74087812] + Random123 v1.7.0
  [e6cf234a] + RandomNumbers v1.6.0
  [6c6a2e73] + Scratch v1.2.1
  [91c51154] + SentinelArrays v1.4.5
  [a2af1166] + SortingAlgorithms v1.2.1
  [90137ffa] + StaticArrays v1.9.7
  [1e83bf80] + StaticArraysCore v1.4.3
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.0
  [a759f4b9] + TimerOutputs v0.5.24
  [013be700] + UnsafeAtomics v0.2.1
  [d80eeb9a] + UnsafeAtomicsLLVM v0.2.1
⌅ [4ee394cb] + CUDA_Driver_jll v0.9.2+0
⌅ [76a88914] + CUDA_Runtime_jll v0.14.1+0
⌅ [dad2f222] + LLVMExtra_jll v0.0.31+0
  [9fa8497b] + Future
  [4af54fe1] + LazyArtifacts
  [8dfed614] + Test
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
7

Does /path-to-home/.julia/artifacts/0b6483311b60bc85d320f64bb66fd2e310916fb1 exist? If so, try removing the directory, which hopefully triggers the artifact getting re-installed.

8

I worked, thank you so much.