# "Your CUDA installation does not provide libcudadevrt" even though it is available in /usr/local/cuda-10.0

**URL:** <https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011>\
**Category:** GPU\
**Tags:** question\
**Created:** [April 4, 2020, 10:54am UTC](https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011 "2020-04-04T10:54:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rmz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rmz/32/9861_2.png) [@rmz](https://discourse.julialang.org/u/rmz)\
**Post date:** [April 4, 2020, 10:54am UTC](https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011/1 "2020-04-04T10:54:29Z")

</div>

I’ve gotten access to a nice machine with a rack full of GPUs running ubuntu. However I can’t get Julia 1.4.0 to use any of the GPUs. Do you have any suggestions for investegations I could do to figure out what’s wrong or things to do to fix the problem?

```julia
$ JULIA_CUDA_VERBOSE=true CUDA_PATH=/usr/local/cuda-10.0 julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.4.0 (2020-03-21)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using CuArrays
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Error: CUDAnative.jl failed to initialize
│ exception =
│ Your CUDA installation does not provide libcudadevrt
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] __init__ () at /storageHD/userHome/rmz/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:146
│ [3] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│ [4] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:781
│ [5] _tryrequire_from_serialized(::Base.PkgId, ::UInt64, ::String) at ./loading.jl:712
│ [6] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:770
│ [7] _require(::Base.PkgId) at ./loading.jl:1006
│ [8] require(::Base.PkgId) at ./loading.jl:927
│ [9] require(::Module, ::Symbol) at ./loading.jl:922
│ [10] eval(::Module, ::Any) at ./boot.jl:331
│ [11] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
│ [12] macro expansion at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:118 [inlined]
│ [13] (::REPL.var"#26#27"{REPL.REPLBackend})() at ./task.jl:358
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:190
┌ Warning: CuArrays.jl did not initialize because CUDAdrv.jl or CUDAnative.jl failed to
└ @ CuArrays ~/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:64

julia> using CUDAdrv; CUDAdrv.CuDevice(0)
CuDevice(0): GeForce RTX 2080 Ti

```

Some more info about the system:

```julia
$ locate libcudadevrt
/usr/local/cuda-10.0/lib64/libcudadevrt.a
$ ls -la /usr/local/cuda-10.0/lib64/libcudadevrt.a
-rw-r--r-- 1 root root 695156 Feb 20 2019 /usr/local/cuda-10.0/lib64/libcudadevrt.a
$ nvidia-smi 
Sat Apr 4 10:51:21 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64 Driver Version: 440.64 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... On | 00000000:04:00.0 Off | N/A |
| 27% 24C P8 1W / 250W | 1108MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 208... On | 00000000:05:00.0 Off | N/A |
| 27% 24C P8 21W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce RTX 208... On | 00000000:06:00.0 Off | N/A |
| 27% 24C P8 21W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 GeForce RTX 208... On | 00000000:07:00.0 Off | N/A |
| 27% 25C P8 1W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 4 GeForce RTX 208... On | 00000000:08:00.0 Off | N/A |
| 27% 24C P8 20W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 5 GeForce RTX 208... On | 00000000:0B:00.0 Off | N/A |
| 27% 24C P8 19W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 6 GeForce RTX 208... On | 00000000:0C:00.0 Off | N/A |
| 27% 24C P8 19W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 7 GeForce RTX 208... On | 00000000:0D:00.0 Off | N/A |
| 27% 23C P8 19W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 8 GeForce RTX 208... On | 00000000:0E:00.0 Off | N/A |
| 27% 25C P8 21W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 9 GeForce RTX 208... On | 00000000:0F:00.0 Off | N/A |
| 27% 25C P8 1W / 250W | 11MiB / 11019MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 19629 C /opt/conda/bin/python 787MiB |
| 0 26698 C ...geHD/userHome/rmz/julia-1.4.0/bin/julia 310MiB |
+-----------------------------------------------------------------------------+

$ dmesg | grep -i ubuntu
[0.000000] Linux version 4.15.0-91-generic (buildd@lgw01-amd64-013) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 (Ubuntu 4.15.0-91.92-generic 4.15.18)

```

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 4, 2020, 12:17pm UTC](https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011/2 "2020-04-04T12:17:05Z")

</div>

Not 100% on this but try adding LD\_LIBRARY\_PATH like:

```julia
LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64 JULIA_CUDA_VERBOSE=true CUDA_PATH=/usr/local/cuda-10.0 julia

```

---

<div class="post-metadata">

**Author:** ![rmz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rmz/32/9861_2.png) [@rmz](https://discourse.julialang.org/u/rmz)\
**Post date:** [April 4, 2020, 8:44pm UTC](https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011/3 "2020-04-04T20:44:01Z")

</div>

> [@pixel27](#):
>
> LD\_LIBRARY\_PATH=/usr/local/cuda-10.0/lib64 JULIA\_CUDA\_VERBOSE=true CUDA\_PATH=/usr/local/cuda-10.0 julia

Hi

Thank you for your answer. I tried doing that (see below) but it seems like that didn’t help immediately. But then I created a fake copy of the cuda home directory, consisting only of symbolic links to the real /usr/local/cuda-10.0 directory, but with an added symbolic link going from lib to lib64. That got me one step ahead, now blocking on. “libcublas.so: wrong ELF class: ELFCLASS64”. … so if you have any ideas about what I could do to get around this I would be very grateful.

```julia
$ LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64 JULIA_CUDA_VERBOSE=true CUDA_PATH=fake_cuda JULIA_CUDA_VERBOSE=true JULIA_DEBUG=CUDAapi julia
               _
   _ _ _(_)_ | Documentation: https://docs.julialang.org
  (_) | (_) (_) |
   _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` | |
  | | |_| | | | (_| | | Version 1.4.0 (2020-03-21)
 _/ |\ __'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |

julia> using CuArrays
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_PATH
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│ locations =
│ 12-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/bin"
│ "/usr/local/cuda-10.0/bin"
│ "/usr/local/sbin"
│ "/usr/local/bin"
│ "/usr/sbin"
│ "/usr/bin"
│ "/sbin"
│ "/bin"
│ "/usr/games"
│ "/snap/bin"
│ "/storageHD/userHome/rmz/julia-1.4.0/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at fake_cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.0.130
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at fake_cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt 
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at fake_cuda/lib
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Warning: Your CUDA installation does not provide the NVTX library, CUDAnative.NVTX will be unavailable
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:153
┌ Debug: Request to look for library cupti
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.0
│ locations =
│ 4-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
│ "fake_cuda/extras/CUPTI"
│ "fake_cuda/extras/CUPTI/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Warning: Your CUDA installation does not provide the CUPTI library, CUDAnative.@code_sass will be unavailable
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:160
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_PATH
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cublas
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcublas.so, libcublas.so.10, libcublas.so.10.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cusparse
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusparse.so, libcusparse.so.10, libcusparse.so.10.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cusolver
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusolver.so, libcusolver.so.10, libcusolver.so.10.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cufft
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcufft.so, libcufft.so.10, libcufft.so.10.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library curand
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcurand.so, libcurand.so.10, libcurand.so.10.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cudnn
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcudnn.so, libcudnn.so.7, libcudnn.so.7.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cutensor
│ locations =
│ 1-element Array{String,1}:
│ "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcutensor.so, libcutensor.so.1, libcutensor.so.1.0
│ locations =
│ 2-element Array{String,1}:
│ "fake_cuda"
│ "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Error: CuArrays.jl failed to initialize
│ exception =
│ could not load library "libcublas"
│ libcublas.so: wrong ELF class: ELFCLASS64
│ Stacktrace:
│ [1] dlopen(::String, ::UInt32; throw_error::Bool) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109
│ [2] dlopen at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│ [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /storageHD/userHome/rmz/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│ [4] macro expansion at /storageHD/userHome/rmz/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│ [5] macro expansion at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/error.jl:65 [inlined]
│ [6] cublasGetProperty at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/libcublas.jl:27 [inlined]
│ [7] cublasGetProperty at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/wrappers.jl:38 [inlined]
│ [8] version() at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/wrappers.jl:42
│ [9] __init__ () at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:99
│ [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│ [11] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:781
│ [12] _require(::Base.PkgId) at ./loading.jl:1006
│ [13] require(::Base.PkgId) at ./loading.jl:927
│ [14] require(::Module, ::Symbol) at ./loading.jl:922
│ [15] eval(::Module, ::Any) at ./boot.jl:331
│ [16] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
│ [17] macro expansion at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:118 [inlined]
│ [18] (::REPL.var"#26#27"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:142

julia> 

```

```julia

```

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [April 5, 2020, 12:30am UTC](https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011/4 "2020-04-05T00:30:25Z")

</div>

Is your Julia 32bit or 64bit? It appears libcublas.so is 64bit, so either you need to install the 64bit version of Julia, or the 32bit version of libcublas.so.

I think you can determine which version of Julia you have by running `versioninfo()`. If the WORD\_SIZE is 64 then you are 64bit, if it’s 32 then 32bit.

---

<div class="post-metadata">

**Author:** ![rmz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rmz/32/9861_2.png) [@rmz](https://discourse.julialang.org/u/rmz)\
**Post date:** [April 5, 2020, 10:19am UTC](https://discourse.julialang.org/t/your-cuda-installation-does-not-provide-libcudadevrt-even-though-it-is-available-in-usr-local-cuda-10-0/37011/5 "2020-04-05T10:19:16Z")

</div>

_Blush_. I installed a 32 bit julia. But … but installing the 64 bit version solved the problem. Ho ho ho, now I have a GPU array 🙂 Thank you so much! Rally appreciate your help.