CUDA could not find an appropiate CUDA runtime to use

lepton01 · April 6, 2023, 11:26pm

Hello there.
Haven’t used CUDA.jl before. I am working with some NN and wanted to see how it would perform on my GPU, if that makes any sense.
After adding CUDA through the Julia Package Manager, using CUDA gives the title error message.

I am on Windows 10, latest Nvidia drivers and verified that my GPU supports CUDA, it is a GTX 1660 SUPER.
nvidia-smi --list-gpus works and recognizes my GPU. driver_version() gives v"12.1.0"
Running with JULIA_DEBUG=all throws:

julia> ENV["JULIA_DEBUG"]="all"
"all"

┌ Debug: Ignoring cache file C:\Users\a\.julia\compiled\v1.8\GPUCompiler\yPwef_EOBFs.ji for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55] since it is does not provide 
desired build_id
└ @ Base loading.jl:2082
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\CEnum\0gyUJ_9lXWA.ji for CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Preferences\pWSk8_EOBFs.ji for Preferences [21216c6a-2e73-6563-6e65-726566657250]    
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\JLLWrappers\7Zgw7_EOBFs.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]    
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\LLVMExtra_jll\R9OeX_JSxXo.ji for LLVMExtra_jll [dad2f222-ce93-54a1-a47d-0025e8a3acab]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\LLVM\e8NBy_JSxXo.ji for LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:806
┌ Debug: Using LLVM 13.0.1 at C:\Users\a\AppData\Local\Programs\Julia-1.8.5\bin\libLLVM-13jl.dll
└ @ LLVM C:\Users\a\.julia\packages\LLVM\TLGyi\src\LLVM.jl:94
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\ExprTools\eM8wu_JSxXo.ji for ExprTools [e2ba6199-217a-4e67-a87a-7c52f15ade04]      
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\TimerOutputs\hd2yD_JSxXo.ji for TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Scratch\ICI1U_JSxXo.ji for Scratch [6c6a2e73-6563-6170-7368-637461726353]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\GPUCompiler\yPwef_JSxXo.ji for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Requires\IyxeS_9lXWA.ji for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Adapt\rUIgN_Cuswp.ji for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:806
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\GPUArraysCore\qiYUe_Cuswp.ji for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\GPUArrays\v5u0T_JSxXo.ji for GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\BFloat16s\iiZ8G_VR7ol.ji for BFloat16s [ab4f0b2a-ad5b-11e8-123f-65d77653426b]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\CUDA_Driver_jll\QJyk7_RRnHg.ji for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\CUDA_Runtime_jll\Hs50y_RRnHg.ji for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\RandomNumbers\pgCpR_VR7ol.ji for RandomNumbers [e6cf234a-135c-5ec9-84dd-332b85af5143]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Random123\1imiM_VR7ol.ji for Random123 [74087812-796a-5b5d-8853-05524746bad3]
└ @ Base loading.jl:806
┌ Debug: Requires conditionally ran code in 0.008615 seconds: `RandomNumbers` detected `Random123`
└ @ Requires C:\Users\a\.julia\packages\RandomNumbers\3pD1N\src\RandomNumbers.jl:38
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Compat\GSFWK_0vkkC.ji for Compat [34da2185-b29b-5c13-b0c7-acf172513d20]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\ChainRulesCore\G6ax7_0vkkC.ji for ChainRulesCore [d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\AbstractFFTs\Di3HZ_JSxXo.ji for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\SnoopPrecompile\TcfDT_EOBFs.ji for SnoopPrecompile [66db9d55-30c0-4569-8b51-7e840670fc0c]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\UnsafeAtomics\OuhNJ_rNYEn.ji for UnsafeAtomics [013be700-e6cd-48c3-b4a1-df204f14c38f]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\Atomix\3LdQ4_rNYEn.ji for Atomix [a9b6321e-bd34-4604-b9c9-b65b8de01458]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\MacroTools\38lnR_9lXWA.ji for MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\StaticArraysCore\Tzw28_9lXWA.ji for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\StaticArrays\yY9vm_rNYEn.ji for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
└ @ Base loading.jl:806
┌ Debug: Requires conditionally ran code in 0.0107621 seconds: `Adapt` detected `StaticArrays`
└ @ Requires C:\Users\a\.julia\packages\Adapt\xviDc\src\Adapt.jl:76
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\UnsafeAtomicsLLVM\auDvx_JSxXo.ji for UnsafeAtomicsLLVM [d80eeb9a-aca5-4d75-85e5-170c8b632249]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\KernelAbstractions\aywHT_JSxXo.ji for KernelAbstractions [63c18a36-062a-441e-b654-da1e3ab1ce7c]
└ @ Base loading.jl:806
┌ Debug: Loading cache file C:\Users\a\.julia\compiled\v1.8\CUDA\oWw5k_RRnHg.ji for CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Base loading.jl:806

Do I need to specify the driver somehow? I read in the docs abput nvml.dll in the Nvidia Corporation folder, but there is not.

lepton01 · April 7, 2023, 12:15am

Fix: use the CUDA.set_runtime_version!(v"12.1.0") function, I did not thought the default would cause the previous issue.

wsshin · May 15, 2023, 6:36pm

I was able to resolve this issue using the above solution. To add more details, I was able to obtain the CUDA version number to use by running nvidia-smi in BASH, which was 11.4 in my case. Then, I did

julia> using CUDA

julia> CUDA.set_runtime_version!(v"11.4")

and restarted REPL. Then the subsequent use of CUDA downloaded the CUDA runtime:

julia> using CUDA
[ Info: Precompiling CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
  Downloaded artifact: CUDA_Runtime

Then I was able to use CUDA normally. Hope this helps people experiencing similar issues.

UPDATE. My system admin says the CUDA version shown in nvidia-smi just means that nvidia-smi program is built with that CUDA version. He says all the CUDA versions available on the system can be obtained by module avail cuda. In my case this showed 11.6 as the most recent version instead of 11.4.

maleadt · May 16, 2023, 11:30am

This shouldn’t have been needed. You’re using a outdated CUDA version now; your driver should support newer. CUDA.jl’s autodetection should work fine; I’m not sure why in some cases it returns cuda+none…

wsshin · May 16, 2023, 2:07pm

@maleadt, FYI, I didn’t have to go through these steps for CUDA.jl 4.0.1 on Julia 1.8: CUDA.jl was able to autodetect whatever artifacts it needed. The described problem started happening only after I started using Julia 1.9 rc: when I added CUDA.jl, it could not find an appropriate CUDA runtime.

Do you think installing an up-to-date CUDA driver would solve the issue? As I added above, the CUDA driver (runtime?) version on our system is certainly outdated (11.6 compared to the most recent 12.1).

maleadt · May 16, 2023, 5:45pm

Strange, Julia 1.9 shouldn’t change anything in this regard. Would be good to figure out what causes this, so if you have a reproducer, please file an issue on CUDA.jl

Updating the CUDA driver shouldn’t matter, as we support both 11.x and 12.x. Unless you need 12.x features, there’s no need to upgrade your driver. And if you’re using enterprise hardware, we do that for you anyway (courtesy of CUDA_Driver_jll).

Wenzhe_Ma · May 20, 2023, 9:07pm

I had the same problem. Your workaround has saved my day.

Nikos_Gianniotis · December 21, 2023, 7:33pm

Same thing here. Thanks for this workaround, I was just about to re-install my Ubuntu as I was suspecting some non-fixable driver issue until I came across your solution. Using Julia 1.10.0-rc2.