"Your CUDA installation does not provide libcudadevrt" even though it is available in /usr/local/cuda-10.0

rmz · April 4, 2020, 10:54am

I’ve gotten access to a nice machine with a rack full of GPUs running ubuntu. However I can’t get Julia 1.4.0 to use any of the GPUs. Do you have any suggestions for investegations I could do to figure out what’s wrong or things to do to fix the problem?

$ JULIA_CUDA_VERBOSE=true CUDA_PATH=/usr/local/cuda-10.0 julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0 (2020-03-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |


julia> using CuArrays
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Error: CUDAnative.jl failed to initialize
│   exception =
│    Your CUDA installation does not provide libcudadevrt
│    Stacktrace:
│     [1] error(::String) at ./error.jl:33
│     [2] __init__() at /storageHD/userHome/rmz/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:146
│     [3] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│     [4] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:781
│     [5] _tryrequire_from_serialized(::Base.PkgId, ::UInt64, ::String) at ./loading.jl:712
│     [6] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:770
│     [7] _require(::Base.PkgId) at ./loading.jl:1006
│     [8] require(::Base.PkgId) at ./loading.jl:927
│     [9] require(::Module, ::Symbol) at ./loading.jl:922
│     [10] eval(::Module, ::Any) at ./boot.jl:331
│     [11] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
│     [12] macro expansion at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:118 [inlined]
│     [13] (::REPL.var"#26#27"{REPL.REPLBackend})() at ./task.jl:358
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:190
┌ Warning: CuArrays.jl did not initialize because CUDAdrv.jl or CUDAnative.jl failed to
└ @ CuArrays ~/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:64

julia>  using CUDAdrv; CUDAdrv.CuDevice(0)
CuDevice(0): GeForce RTX 2080 Ti

Some more info about the system:

$ locate libcudadevrt
/usr/local/cuda-10.0/lib64/libcudadevrt.a
$ ls -la /usr/local/cuda-10.0/lib64/libcudadevrt.a
-rw-r--r-- 1 root root 695156 Feb 20  2019 /usr/local/cuda-10.0/lib64/libcudadevrt.a
$ nvidia-smi 
Sat Apr  4 10:51:21 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:04:00.0 Off |                  N/A |
| 27%   24C    P8     1W / 250W |   1108MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  On   | 00000000:05:00.0 Off |                  N/A |
| 27%   24C    P8    21W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  On   | 00000000:06:00.0 Off |                  N/A |
| 27%   24C    P8    21W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  On   | 00000000:07:00.0 Off |                  N/A |
| 27%   25C    P8     1W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce RTX 208...  On   | 00000000:08:00.0 Off |                  N/A |
| 27%   24C    P8    20W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce RTX 208...  On   | 00000000:0B:00.0 Off |                  N/A |
| 27%   24C    P8    19W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce RTX 208...  On   | 00000000:0C:00.0 Off |                  N/A |
| 27%   24C    P8    19W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce RTX 208...  On   | 00000000:0D:00.0 Off |                  N/A |
| 27%   23C    P8    19W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   8  GeForce RTX 208...  On   | 00000000:0E:00.0 Off |                  N/A |
| 27%   25C    P8    21W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   9  GeForce RTX 208...  On   | 00000000:0F:00.0 Off |                  N/A |
| 27%   25C    P8     1W / 250W |     11MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     19629      C   /opt/conda/bin/python                        787MiB |
|    0     26698      C   ...geHD/userHome/rmz/julia-1.4.0/bin/julia   310MiB |
+-----------------------------------------------------------------------------+

$ dmesg | grep -i ubuntu
[    0.000000] Linux version 4.15.0-91-generic (buildd@lgw01-amd64-013) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 (Ubuntu 4.15.0-91.92-generic 4.15.18)

pixel27 · April 4, 2020, 12:17pm

Not 100% on this but try adding LD_LIBRARY_PATH like:

LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64 JULIA_CUDA_VERBOSE=true CUDA_PATH=/usr/local/cuda-10.0 julia

rmz · April 4, 2020, 8:44pm

Hi

Thank you for your answer. I tried doing that (see below) but it seems like that didn’t help immediately. But then I created a fake copy of the cuda home directory, consisting only of symbolic links to the real /usr/local/cuda-10.0 directory, but with an added symbolic link going from lib to lib64. That got me one step ahead, now blocking on. “libcublas.so: wrong ELF class: ELFCLASS64”. … so if you have any ideas about what I could do to get around this I would be very grateful.

$ LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64 JULIA_CUDA_VERBOSE=true CUDA_PATH=fake_cuda JULIA_CUDA_VERBOSE=true JULIA_DEBUG=CUDAapi   julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.0 (2020-03-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using CuArrays
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_PATH
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│   locations =
│    12-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/bin"
│     "/usr/local/cuda-10.0/bin"
│     "/usr/local/sbin"
│     "/usr/local/bin"
│     "/usr/sbin"
│     "/usr/bin"
│     "/sbin"
│     "/bin"
│     "/usr/games"
│     "/snap/bin"
│     "/storageHD/userHome/rmz/julia-1.4.0/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at fake_cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.0.130
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at fake_cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt 
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at fake_cuda/lib
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Warning: Your CUDA installation does not provide the NVTX library, CUDAnative.NVTX will be unavailable
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:153
┌ Debug: Request to look for library cupti
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.0
│   locations =
│    4-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
│     "fake_cuda/extras/CUPTI"
│     "fake_cuda/extras/CUPTI/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Warning: Your CUDA installation does not provide the CUPTI library, CUDAnative.@code_sass will be unavailable
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:160
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_PATH
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cublas
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcublas.so, libcublas.so.10, libcublas.so.10.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cusparse
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusparse.so, libcusparse.so.10, libcusparse.so.10.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cusolver
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusolver.so, libcusolver.so.10, libcusolver.so.10.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cufft
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcufft.so, libcufft.so.10, libcufft.so.10.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library curand
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcurand.so, libcurand.so.10, libcurand.so.10.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cudnn
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcudnn.so, libcudnn.so.7, libcudnn.so.7.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for library cutensor
│   locations =
│    1-element Array{String,1}:
│     "fake_cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcutensor.so, libcutensor.so.1, libcutensor.so.1.0
│   locations =
│    2-element Array{String,1}:
│     "fake_cuda"
│     "fake_cuda/lib"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Error: CuArrays.jl failed to initialize
│   exception =
│    could not load library "libcublas"
│    libcublas.so: wrong ELF class: ELFCLASS64
│    Stacktrace:
│     [1] dlopen(::String, ::UInt32; throw_error::Bool) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109
│     [2] dlopen at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│     [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /storageHD/userHome/rmz/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│     [4] macro expansion at /storageHD/userHome/rmz/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│     [5] macro expansion at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/error.jl:65 [inlined]
│     [6] cublasGetProperty at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/libcublas.jl:27 [inlined]
│     [7] cublasGetProperty at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/wrappers.jl:38 [inlined]
│     [8] version() at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/blas/wrappers.jl:42
│     [9] __init__() at /storageHD/userHome/rmz/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:99
│     [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│     [11] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:781
│     [12] _require(::Base.PkgId) at ./loading.jl:1006
│     [13] require(::Base.PkgId) at ./loading.jl:927
│     [14] require(::Module, ::Symbol) at ./loading.jl:922
│     [15] eval(::Module, ::Any) at ./boot.jl:331
│     [16] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
│     [17] macro expansion at /buildworker/worker/package_linux32/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:118 [inlined]
│     [18] (::REPL.var"#26#27"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:142

julia>

pixel27 · April 5, 2020, 12:30am

Is your Julia 32bit or 64bit? It appears libcublas.so is 64bit, so either you need to install the 64bit version of Julia, or the 32bit version of libcublas.so.

I think you can determine which version of Julia you have by running versioninfo(). If the WORD_SIZE is 64 then you are 64bit, if it’s 32 then 32bit.

rmz · April 5, 2020, 10:19am

Blush. I installed a 32 bit julia. But … but installing the 64 bit version solved the problem. Ho ho ho, now I have a GPU array Thank you so much! Rally appreciate your help.