Error using system CUDA in CUDA.jl

Specific Domains GPU
1

Hi all, I’m trying to install CUDA.jl using my system’s CUDA toolkit. The installation completes, but I get errors when I try to test it by running CUDA.version().

I set JULIA_CUDA_USE_BINARYBUILDER=false and ensured that CUDA_ROOT is set properly. When I run CUDA.version(), I get the following. Actually, the stacktrace is really huge, but this message was repeated several times.

I am using CUDA.jl v3.8.3 with a local installation of CUDA 10.1

┌ Debug: Could not use forward compatibility package
│   exception =
│    CUDA error: system has unsupported display driver / cuda driver combination (code 803, ERROR_SYSTEM_DRIVER_MISMATCH)
│    Stacktrace:
│         [1] throw_api_error(res::CUDA.cudaError_enum)
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/error.jl:91
│         [2] macro expansion
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/error.jl:101 [inlined]
│         [3] (::CUDA.var"#do_init#3")(driver::String)
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/CUDAdrv.jl:88
│         [4] find_libcuda()
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/CUDAdrv.jl:116
│         [5] libcuda()
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/CUDAdrv.jl:141
│         [6] macro expansion
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/libcuda.jl:5 [inlined]
│         [7] macro expansion
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/error.jl:97 [inlined]
│         [8] cuGetErrorString
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/utils/call.jl:26 [inlined]
│         [9] description
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/error.jl:53 [inlined]
│        [10] showerror(io::IOContext{IOBuffer}, err::CUDA.CuError)
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/error.jl:60
│        [11] showerror(io::IOContext{IOBuffer}, ex::CUDA.CuError, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}; backtrace::Bool)
│           @ Base ./errorshow.jl:88
│        [12] showvalue(io::IOContext{IOBuffer}, e::Tuple{CUDA.CuError, Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}})
│           @ Logging /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/Logging/src/ConsoleLogger.jl:56
│        [13] handle_message(logger::Logging.ConsoleLogger, level::Base.CoreLogging.LogLevel, message::Any, _module::Any, group::Any, id::Any, filepath::Any, line::Any; kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
│           @ Logging /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/Logging/src/ConsoleLogger.jl:134
│        [14] macro expansion
│           @ ./logging.jl:365 [inlined]
│        [15] find_libcuda()
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/CUDAdrv.jl:119
│    --- the last 11 lines are repeated 1604 more times ---
│     [17660] libcuda()
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/CUDAdrv.jl:141
│     [17661] macro expansion
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/libcuda.jl:23 [inlined]
│     [17662] macro expansion
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/error.jl:97 [inlined]
│     [17663] cuDriverGetVersion
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/utils/call.jl:26 [inlined]
│     [17664] version()
│           @ CUDA /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/lib/cudadrv/version.jl:24
│     [17665] top-level scope
│           @ REPL[2]:1
│     [17666] top-level scope
│           @ /usr/local/pace-apps/manual/packages/julia/1.7.2/gcc-8.3.0/pace/packages/CUDA/Axzxe/src/initialization.jl:52
│     [17667] eval
│           @ ./boot.jl:373 [inlined]
│     [17668] eval_user_input(ast::Any, backend::REPL.REPLBackend)
│           @ REPL /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/REPL/src/REPL.jl:150
│     [17669] repl_backend_loop(backend::REPL.REPLBackend)
│           @ REPL /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/REPL/src/REPL.jl:246
│     [17670] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
│           @ REPL /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/REPL/src/REPL.jl:231
│     [17671] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
│           @ REPL /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/REPL/src/REPL.jl:364
│     [17672] run_repl(repl::REPL.AbstractREPL, consumer::Any)
│           @ REPL /storage/coda-apps/test/manual/packages/julia/1.7.2/gcc-8.3.0/share/julia/stdlib/v1.7/REPL/src/REPL.jl:351
│     [17673] (::Base.var"#930#932"{Bool, Bool, Bool})(REPL::Module)
│           @ Base ./client.jl:394
│     [17674] #invokelatest#2
│           @ ./essentials.jl:716 [inlined]
│     [17675] invokelatest
│           @ ./essentials.jl:714 [inlined]
│     [17676] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
│           @ Base ./client.jl:379
│     [17677] exec_options(opts::Base.JLOptions)
│           @ Base ./client.jl:
2

The 803 is generated outside Julia by the CUDA subsystem

3

The warning that CUDA.jl isn’t able to load the forward compatibility driver is a warning, nothing more. It isn’t causing any errors. If you have other issues, look for another exception being thrown. You can also try the master branch of CUDA.jl, where error messages have been slightly improved.

4

I just went ahead and let Julia install its own CUDA drivers (which were 11.5 for me), and now it’s working great. I don’t imagine there’s much benefit to using my system’s CUDA 10.5 installation if it’s not compatible.