Failure to run CUDA.jl

Specific Domains GPU
1

Hi,
I just installed julia 1.5.3 and CUDA.jl in an office computer that runs ubuntu 18.04 and that has old UCDA drivers (from a couple of years ago). I can install CUDA.jl but when I try to use it, if fails to start and spits all this

  _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using CUDA

julia> CUDA.zeros(10)
┌ Error: Recursion during initialization of CUDA.jl
└ @ CUDA ~/.julia/packages/CUDA/FlHUF/src/initialization.jl:38
┌ Error: Error during initialization of CUDA.jl
│   exception =
│    CUDA error (code 100, CUDA_ERROR_NO_DEVICE)
│    Stacktrace:
│     [1] throw_api_error(::CUDA.cudaError_enum) at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/error.jl:97
│     [2] __configure__() at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/initialization.jl:93
│     [3] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/initialization.jl:30 [inlined]
│     [4] macro expansion at ./lock.jl:183 [inlined]
│     [5] _functional(::Bool) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/initialization.jl:26
│     [6] functional(::Bool) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/initialization.jl:19
│     [7] libcuda() at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/initialization.jl:47
│     [8] (::CUDA.var"#690#cache_fptr!#9")() at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/utils/call.jl:31
│     [9] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/utils/call.jl:39 [inlined]
│     [10] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/libcuda.jl:29 [inlined]
│     [11] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/error.jl:102 [inlined]
│     [12] cuDeviceGet(::Base.RefValue{Int32}, ::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/utils/call.jl:93
│     [13] CuDevice(::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/devices.jl:25
│     [14] initialize_thread(::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/state.jl:121
│     [15] prepare_cuda_call() at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/state.jl:80
│     [16] device at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/state.jl:227 [inlined]
│     [17] alloc at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/pool.jl:293 [inlined]
│     [18] CuArray{Float32,1}(::UndefInitializer, ::Tuple{Int64}) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:20
│     [19] CuArray at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:76 [inlined]
│     [20] CuArray at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:77 [inlined]
│     [21] zeros at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:348 [inlined]
│     [22] zeros(::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:350
│     [23] top-level scope at REPL[2]:1
│     [24] eval(::Module, ::Any) at ./boot.jl:331
│     [25] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
│     [26] repl_backend_loop(::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
│     [27] start_repl_backend(::REPL.REPLBackend, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
│     [28] run_repl(::REPL.AbstractREPL, ::Any; backend_on_current_task::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
│     [29] run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
│     [30] (::Base.var"#807#809"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:399
│     [31] #invokelatest#1 at ./essentials.jl:710 [inlined]
│     [32] invokelatest at ./essentials.jl:709 [inlined]
│     [33] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:383
│     [34] exec_options(::Base.JLOptions) at ./client.jl:313
│     [35] _start() at ./client.jl:506
└ @ CUDA ~/.julia/packages/CUDA/FlHUF/src/initialization.jl:34
ERROR: CUDA.jl did not successfully initialize, and is not usable.
If you did not see any other error message, try again in a new session
with the JULIA_DEBUG environment variable set to 'CUDA'.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] libcuda() at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/initialization.jl:48
 [3] (::CUDA.var"#690#cache_fptr!#9")() at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/utils/call.jl:31
 [4] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/utils/call.jl:39 [inlined]
 [5] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/libcuda.jl:29 [inlined]
 [6] macro expansion at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/error.jl:102 [inlined]
 [7] cuDeviceGet(::Base.RefValue{Int32}, ::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/utils/call.jl:93
 [8] CuDevice(::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/lib/cudadrv/devices.jl:25
 [9] initialize_thread(::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/state.jl:121
 [10] prepare_cuda_call() at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/state.jl:80
 [11] device at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/state.jl:227 [inlined]
 [12] alloc at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/pool.jl:293 [inlined]
 [13] CuArray{Float32,1}(::UndefInitializer, ::Tuple{Int64}) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:20
 [14] CuArray at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:76 [inlined]
 [15] CuArray at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:77 [inlined]
 [16] zeros at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:348 [inlined]
 [17] zeros(::Int64) at /home/mazzanti/.julia/packages/CUDA/FlHUF/src/array.jl:350
 [18] top-level scope at REPL[2]:1

This is not the first time I’ve seen it but can’t figure out how to proceed.
I would appreciate it if somebody can help me about how to proceed. I don’t even seem able to remove the old CUDA toolkit and start a fresh installation…

Thankx,

Ferran.

2

Your CUDA set-up is broken, as it reports CUDA_ERROR_NO_DEVICE. Nothing we can do about that. Possible reason: at some point NVIDIA removes device support from their latest drivers, and you need to use the legacy one. Maybe you have the wrong driver installed.

3

Ok this is the second computer with this problem. Both ubuntu 18.04, with Julia 1.4.2 & CuArrays/CUDAdrv working fine a year ago.
Uograded to julia 1.5.3 and CUDA.jl and change nothing else (no drivers or hardware) and now it does not work. If the same drivers recognized the same hardware before… why not now?
What is the legacy driver, and how do I get it?
Thanks,
Ferran

4

This is very unlikely, something else is probably going on. For completeness, this error is thrown by the CUDA driver during its initialization. CUDA.jl has done nothing at this point, so no change there could influence whether this error gets thrown or not. I suggest you verify your set-up, e.g., see if nvidia-smi reports the GPU, check dmesg for NVIDIA-related errors, try the deviceQuery sample from the toolkit, etc. It’s much more likely that you, e.g., upgraded your kernel and the NVIDIA module didn’t recompile (I can’t imagine you not having installed any update over the course of the last year).