CUDAnative.jl failed to initialize

Hi,
I’ve been using CUDAnative, CUDAdrv and CuArrays extensively in my ubuntu box for few months now, but somehow after an update of julia packages (which included these) I can no longer use them

using CUDAnative
> [ Info: CUDAnative.jl failed to initialize, GPU functionality unavailable (set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)

and that’s the end of the story, no GPU fun whatsoever

using CuArrays
x = rand(10)
> 10-element Array{Float64,1}:
>  0.1377563892159568 
>  0.5731161511643168 
>  0.7507422246081985 
>  0.07795296478968727
>  0.2584849271207008 
>  0.12356241198243967
>  0.7063796449104687 
>  0.6341081262436765 
>  0.5587673282007684 
> 0.06954422023043616
CuArray(x)
> ERROR: CUDA error: invalid device context (code 201, ERROR_INVALID_CONTEXT)
Stacktrace:
 [1] throw_api_error(::CUDAdrv.cudaError_enum) at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/error.jl:131
 [2] macro expansion at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/error.jl:144 [inlined]
 [3] cuMemAlloc_v2 at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/libcuda.jl:313 [inlined]
 [4] alloc(::Type{CUDAdrv.Mem.DeviceBuffer}, ::Int64) at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/memory.jl:70
 [5] macro expansion at /home/mazzanti/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:228 [inlined]
 [6] actual_alloc(::Int64) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory.jl:60
 [7] actual_alloc at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:55 [inlined]
 [8] macro expansion at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:198 [inlined]
 [9] macro expansion at /home/mazzanti/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:228 [inlined]
 [10] pool_alloc(::Int64, ::Int64) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:197
 [11] (::CuArrays.BinnedPool.var"#12#13"{Int64,Int64,Set{CuArrays.BinnedPool.Block},Array{CuArrays.BinnedPool.Block,1}})() at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:293
 [12] lock(::CuArrays.BinnedPool.var"#12#13"{Int64,Int64,Set{CuArrays.BinnedPool.Block},Array{CuArrays.BinnedPool.Block,1}}, ::ReentrantLock) at ./lock.jl:151
 [13] alloc(::Int64) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:292
 [14] macro expansion at /home/mazzanti/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:228 [inlined]
 [15] macro expansion at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory.jl:159 [inlined]
 [16] macro expansion at ./util.jl:212 [inlined]
 [17] alloc at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory.jl:158 [inlined]
 [18] CuArray{Float64,1,P} where P(::UndefInitializer, ::Tuple{Int64}) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/array.jl:92
 [19] CuArray at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/array.jl:176 [inlined]
 [20] CuArray(::Array{Float64,1}) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/array.jl:187
 [21] top-level scope at REPL[4]:1

any idea of what is going on, and how to fix that? This is ubuntu mate 18.04, and julia 1.3.0

Thx,

Ferran

Still I need to work, so if anything goes wrong, could you please tell me how to revert to a previous version that worked? That happens after an update I did today…
Thanks again,
Ferran.

(set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)

Did you try that?

uhh… no. I undetstood this was a flag to avoid the message from appearing on the screen, not to solve it…

Ok, did that and the system told me that I was missing the CPUIT library (never happened that before). Went to the nVidia website and followed the procedure to install it written there. I then rebooted my machine and now I can’t even login in my linux box, it complains about ‘Failed to start Detect the availabl… and deal with the system changes. See systemvtl status gpu-manager.service for details’. I can not provide more details as, as I said, I can not even logim into my machibe anymore…

Any help, please?

Ok, fixed… just had to reinstall ubuntu from scratch, the toolkit CUDA driver and julia. Just a pece of cake :frowning:

We now depend on the CUPTI library for some functionality, as you noticed. It should have been installed alongside with the CUDA toolkit though. I might have a look at improving the error message.

I have CUDA 10.1. The same error popped up today after updating my packages. I am on Windows 10 on my side. So no CUPTI either. Updating to 10.2 did not solve the problem, I uninstalled the previous version then I did the complete installation of the new one.

I have the same issue using windows 10 and CUDA 10.2.

I have cupti64_102.dll in the directory C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\extras\CUPTI\lib64

It seems like CUDAnative does not try that path:

┌ Debug: Looking for library cupti64, cupti, cupti64_10, cupti64_102
│   locations =
│    3-element Array{String,1}:
│     "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI"
│     "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI\\bin"
│     "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI\\bin\\x64"
└ @ CUDAapi E:\Programs\julia\.julia\packages\CUDAapi\3JKC9\src\CUDAapi.jl:8
[ Info: CUDAnative.jl failed to initialize, GPU functionality unavailable (set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)

It appears as if I can work around the issue by manually loading cupti64_102:

julia> using Libdl

julia> ldir = "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI\\lib64";

julia> Libdl.dlopen(joinpath(ldir, "cupti64_102"), Libdl.RTLD_LAZY;throw_error=false)
Ptr{Nothing} @0x00007ffb79bd0000

julia> using CuArrays

julia> cu([1,2,3])
3-element CuArray{Float32,1,Nothing}:
 1.0
 2.0
 3.0

Thanks for the details, I’ll make CUPTI optional for now and also add those search paths to CUDAapi.

Should be fixed: https://github.com/JuliaGPU/CUDAnative.jl/issues/549#issuecomment-584056173

In Ubuntu 18.04 you can install CUPTI by:

sudo apt install  libcupti-dev libcupti-doc libcupti9.1