Can't get CUDA.jl to see cuda in Ubuntu

taotree · July 8, 2023, 12:50am

I’m on a Dell Laptop with an NVIDIA RTX A2000 8GB Laptop. I’ve installed nvidia-driver-535, nvidia-cuda-toolkit, and cuda (and probably other things) via apt.

I can’t seem to find a particular error. I run with JULIA_DEBUG=all in env:
julia --project -e "using CUDA" 2> output.txt

┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/CEnum/0gyUJ_nEBbB.so for CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Preferences/pWSk8_nEBbB.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/JLLWrappers/7Zgw7_nEBbB.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/LLVMExtra_jll/R9OeX_nEBbB.so for LLVMExtra_jll [dad2f222-ce93-54a1-a47d-0025e8a3acab]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/LLVM/e8NBy_nEBbB.so for LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1051
┌ Debug: Using LLVM 14.0.6 at /home/jshellman/app/julia-1.9.2/bin/../lib/julia/libLLVM-14jl.so
└ @ LLVM ~/.julia/packages/LLVM/RFlwq/src/LLVM.jl:96
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/ExprTools/eM8wu_nEBbB.so for ExprTools [e2ba6199-217a-4e67-a87a-7c52f15ade04]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/TimerOutputs/hd2yD_nEBbB.so for TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Scratch/ICI1U_nEBbB.so for Scratch [6c6a2e73-6563-6170-7368-637461726353]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/GPUCompiler/yPwef_nEBbB.so for GPUCompiler [61eb1bfa-7361-4325-ad38-22787b887f55]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Adapt/rUIgN_nEBbB.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Reexport/bTpYr_nEBbB.so for Reexport [189a3867-3050-52da-a836-e630ba90ab69]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/GPUArraysCore/qiYUe_nEBbB.so for GPUArraysCore [46192b85-c4d5-4398-a991-12ede77f4527]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Statistics/ERcPL_nEBbB.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/GPUArrays/v5u0T_nEBbB.so for GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Requires/IyxeS_nEBbB.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/BFloat16s/iiZ8G_nEBbB.so for BFloat16s [ab4f0b2a-ad5b-11e8-123f-65d77653426b]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_nEBbB.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1051
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/3xFy2/src/wrappers/x86_64-linux-gnu.jl:87
┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/3xFy2/src/wrappers/x86_64-linux-gnu.jl:105
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/CUDA_Runtime_jll/Hs50y_ZdyVv.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/RandomNumbers/pgCpR_nEBbB.so for RandomNumbers [e6cf234a-135c-5ec9-84dd-332b85af5143]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Random123/1imiM_nEBbB.so for Random123 [74087812-796a-5b5d-8853-05524746bad3]
└ @ Base loading.jl:1051
┌ Debug: Requires conditionally ran code in 0.00213054 seconds: `RandomNumbers` detected `Random123`
└ @ Requires ~/.julia/packages/RandomNumbers/3pD1N/src/RandomNumbers.jl:38
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/AbstractFFTs/Di3HZ_nEBbB.so for AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/PrecompileTools/AQ9Mk_nEBbB.so for PrecompileTools [aea7be01-6a6a-4083-8856-8a6e6704d82a]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/UnsafeAtomics/OuhNJ_nEBbB.so for UnsafeAtomics [013be700-e6cd-48c3-b4a1-df204f14c38f]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/Atomix/3LdQ4_nEBbB.so for Atomix [a9b6321e-bd34-4604-b9c9-b65b8de01458]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/MacroTools/38lnR_nEBbB.so for MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/StaticArraysCore/Tzw28_nEBbB.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/StaticArrays/yY9vm_nEBbB.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/AdaptStaticArraysExt/9bCdf_nEBbB.so for AdaptStaticArraysExt [e1699a77-9e31-5da8-bb3e-0a796f95f0a0]
└ @ Base loading.jl:1051
┌ Debug: Extension AdaptStaticArraysExt of Adapt loaded
└ @ Base loading.jl:1256
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/StaticArraysStatisticsExt/EfhbW_nEBbB.so for StaticArraysStatisticsExt [3453fcb0-342e-5e4a-a93b-f2e19ba3ba0f]
└ @ Base loading.jl:1051
┌ Debug: Extension StaticArraysStatisticsExt of StaticArrays loaded
└ @ Base loading.jl:1256
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/UnsafeAtomicsLLVM/auDvx_nEBbB.so for UnsafeAtomicsLLVM [d80eeb9a-aca5-4d75-85e5-170c8b632249]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/KernelAbstractions/aywHT_nEBbB.so for KernelAbstractions [63c18a36-062a-441e-b654-da1e3ab1ce7c]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /home/jshellman/.julia/compiled/v1.9/CUDA/oWw5k_nEBbB.so for CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Base loading.jl:1051
┌ Error: CUDA.jl could not find an appropriate CUDA runtime to use.
│ 
│ This can have several reasons:
│ * you are using an unsupported platform: this version of CUDA.jl
│   only supports Linux (x86_64, aarch64, ppc64le) and Windows (x86_64),
│   while your platform was identified as x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+none-julia_version+1.9.2;
│ * you precompiled CUDA.jl in an environment where the CUDA driver
│   was not available (i.e., a container, or an HPC login node).
│   in that case, you need to specify which CUDA version to use
│   by calling `CUDA.set_runtime_version!`;
│ * you requested use of a local CUDA toolkit, but not all
│   required components were discovered. try running with
│   JULIA_DEBUG=all in your environment for more details.
│ 
│ For more details, refer to the CUDA.jl documentation at
│ https://cuda.juliagpu.org/stable/installation/overview/
└ @ CUDA ~/.julia/packages/CUDA/tVtYo/src/initialization.jl:82

I tried setting runtime to 12.2, didn’t help. I see the “cuda+none” there in the output, but not sure why or how to fix it.

Here’s the nvidia-smi output:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A2000 8GB Lap...    Off | 00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8               4W /  35W |     85MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      3784      G   /usr/lib/xorg/Xorg                           74MiB |
|    0   N/A  N/A      4393      G   /usr/bin/gnome-shell                          5MiB |
+---------------------------------------------------------------------------------------+

ufechner7 · July 8, 2023, 1:59am

There is no need to install cuda with apt when you use Julia. Julia installs all required libraries itself.

mkitti · July 8, 2023, 2:14am

I think you need to install some up to date Nvidia drivers though.

taotree · July 8, 2023, 3:48am

What makes you say that? I think I’m running 535 which I thought was the most recent major version.

mkitti · July 8, 2023, 4:30am

Something has gone wrong with. Maybe it’s the opposite problem and it’s too new?

github.com

JuliaBinaryWrappers/CUDA_Runtime_jll.jl/blob/main/.pkg/platform_augmentation.jl

using Base.BinaryPlatforms

using Base: thismajor, thisminor

using Libdl

try
    using CUDA_Driver_jll
catch err
    # during initial package installation, CUDA_Driver_jll may not be available.
    # in that case, we just won't select an artifact (unless the user has set a preference).
    # see also: https://github.com/JuliaLang/Pkg.jl/issues/3225
end

# Can't use Preferences for the same reason
const CUDA_Runtime_jll_uuid = Base.UUID("76a88914-d11a-5bdc-97e0-2f5a05c973a2")
const preferences = Base.get_preferences(CUDA_Runtime_jll_uuid)
Base.record_compiletime_preference(CUDA_Runtime_jll_uuid, "version")

# returns the value for the "cuda" tag we should use in the platform.

This file has been truncated. show original

and cannot quite match any of the artifacts?

github.com

JuliaBinaryWrappers/CUDA_Runtime_jll.jl/blob/main/Artifacts.toml

[[CUDA_Runtime]]
arch = "aarch64"
cuda = "10.2"
git-tree-sha1 = "8d8d25ee2c7d9b9db0e1e67cb5477de21468bc75"
lazy = true
libc = "glibc"
os = "linux"

    [[CUDA_Runtime.download]]
    sha256 = "58e75870e8b1e67cbd7bd1611049153ee82851ee6aaffa51206aa63d20d3fe4c"
    url = "https://github.com/JuliaBinaryWrappers/CUDA_Runtime_jll.jl/releases/download/CUDA_Runtime-v0.2.4+1/CUDA_Runtime.v0.2.4.aarch64-linux-gnu-cuda+10.2.tar.gz"
[[CUDA_Runtime]]
arch = "x86_64"
cuda = "10.2"
git-tree-sha1 = "617b9fe8c0cdfdba72e2295b2aaf4e2b091632b9"
lazy = true
libc = "glibc"
os = "linux"

    [[CUDA_Runtime.download]]

This file has been truncated. show original

Maybe try something like this:

julia> using CUDA

julia> CUDA.set_runtime_version!(v"11.8")

That came from

https://cuda.juliagpu.org/stable/installation/overview/#CUDA-toolkit

Otherwise, @maleadt would probably know best.

gdalle · July 8, 2023, 6:28am

I remember having had to downgrade my driver for CUDA.jl to work

maleadt · July 8, 2023, 8:21am

That shouldn’t ever be required.

What does nvidia-smi report?

taotree · July 8, 2023, 12:58pm

I added the nvidia-smi output to the original post.

taotree · July 8, 2023, 3:16pm

I appreciate the help and suggestions. I think I had just gotten the drivers into a bad state or something. I did a clean reinstall of Ubuntu on the machine, used the standard Ubuntu “Additional Software” way of selecting the nvidia driver, and now it’s working.

maleadt · July 8, 2023, 8:36pm

Maybe you installed/precompiled CUDA.jl before you had the NVIDIA driver set-up? In that case, this would be Runtime not re-selected after driver upgrade · Issue #1877 · JuliaGPU/CUDA.jl · GitHub

sob · August 28, 2023, 8:28pm

For CUDA under Linux, should one use nouveau or the proprietary NVidia driver? Does it make a difference as far as CUDA is concerned?

maleadt · August 28, 2023, 8:42pm

You need the NVIDIA proprietary driver, not nouveau. In fact, you should blacklist nouveau, as loading it will break the NVIDIA driver (such a blacklist is provided automatically by most distros).

For the sake of accuracy: there’s also an open driver by NVIDIA nowadays, and that driver also works with CUDA.jl. However, it still requires nvidia-utils providing libcuda.so, and that part is still closed-source.

sob · August 29, 2023, 10:42am

Thank you. I was curious as my 1.9 install is fine, however my 1.10-beta2 throws a warning during install (which doesn’t prevent CUDA.functional() from returning true):

  1 dependency had warnings during precompilation:
┌ FluxCUDAcuDNNExt [a1ab010e-6f71-51d6-acc8-fb8bfa4c0f92]
│  [pid 58614] waiting for IO to finish:
│   TYPE[FD/PID]       @UV_HANDLE_T->DATA
│   timer              @0x299b690->0x7f567b156440
└

ufechner7 · August 29, 2023, 11:44am

This is to be expected and will hopefully be fixed in the final version of 1.10.

sob · October 4, 2023, 1:50pm

I was hoping for beta3 to fix this but I’ll have to be more patient!

ufechner7 · October 4, 2023, 4:41pm

sob · October 4, 2023, 4:55pm

Thanks for the heads up