CUDAnative.jl failed to initialize

Specific Domains GPU
1

Hi,
I’ve been using CUDAnative, CUDAdrv and CuArrays extensively in my ubuntu box for few months now, but somehow after an update of julia packages (which included these) I can no longer use them

using CUDAnative
> [ Info: CUDAnative.jl failed to initialize, GPU functionality unavailable (set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)

and that’s the end of the story, no GPU fun whatsoever

using CuArrays
x = rand(10)
> 10-element Array{Float64,1}:
>  0.1377563892159568 
>  0.5731161511643168 
>  0.7507422246081985 
>  0.07795296478968727
>  0.2584849271207008 
>  0.12356241198243967
>  0.7063796449104687 
>  0.6341081262436765 
>  0.5587673282007684 
> 0.06954422023043616
CuArray(x)
> ERROR: CUDA error: invalid device context (code 201, ERROR_INVALID_CONTEXT)
Stacktrace:
 [1] throw_api_error(::CUDAdrv.cudaError_enum) at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/error.jl:131
 [2] macro expansion at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/error.jl:144 [inlined]
 [3] cuMemAlloc_v2 at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/libcuda.jl:313 [inlined]
 [4] alloc(::Type{CUDAdrv.Mem.DeviceBuffer}, ::Int64) at /home/mazzanti/.julia/packages/CUDAdrv/b1mvw/src/memory.jl:70
 [5] macro expansion at /home/mazzanti/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:228 [inlined]
 [6] actual_alloc(::Int64) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory.jl:60
 [7] actual_alloc at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:55 [inlined]
 [8] macro expansion at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:198 [inlined]
 [9] macro expansion at /home/mazzanti/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:228 [inlined]
 [10] pool_alloc(::Int64, ::Int64) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:197
 [11] (::CuArrays.BinnedPool.var"#12#13"{Int64,Int64,Set{CuArrays.BinnedPool.Block},Array{CuArrays.BinnedPool.Block,1}})() at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:293
 [12] lock(::CuArrays.BinnedPool.var"#12#13"{Int64,Int64,Set{CuArrays.BinnedPool.Block},Array{CuArrays.BinnedPool.Block,1}}, ::ReentrantLock) at ./lock.jl:151
 [13] alloc(::Int64) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory/binned.jl:292
 [14] macro expansion at /home/mazzanti/.julia/packages/TimerOutputs/7Id5J/src/TimerOutput.jl:228 [inlined]
 [15] macro expansion at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory.jl:159 [inlined]
 [16] macro expansion at ./util.jl:212 [inlined]
 [17] alloc at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/memory.jl:158 [inlined]
 [18] CuArray{Float64,1,P} where P(::UndefInitializer, ::Tuple{Int64}) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/array.jl:92
 [19] CuArray at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/array.jl:176 [inlined]
 [20] CuArray(::Array{Float64,1}) at /home/mazzanti/.julia/packages/CuArrays/AIMph/src/array.jl:187
 [21] top-level scope at REPL[4]:1

any idea of what is going on, and how to fix that? This is ubuntu mate 18.04, and julia 1.3.0

Thx,

Ferran

2

Still I need to work, so if anything goes wrong, could you please tell me how to revert to a previous version that worked? That happens after an update I did today…
Thanks again,
Ferran.

3 
(set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)

Did you try that?

4

uhh… no. I undetstood this was a flag to avoid the message from appearing on the screen, not to solve it…

5

Ok, did that and the system told me that I was missing the CPUIT library (never happened that before). Went to the nVidia website and followed the procedure to install it written there. I then rebooted my machine and now I can’t even login in my linux box, it complains about ‘Failed to start Detect the availabl… and deal with the system changes. See systemvtl status gpu-manager.service for details’. I can not provide more details as, as I said, I can not even logim into my machibe anymore…

Any help, please?

6

Ok, fixed… just had to reinstall ubuntu from scratch, the toolkit CUDA driver and julia. Just a pece of cake :frowning:

1 Like
7

We now depend on the CUPTI library for some functionality, as you noticed. It should have been installed alongside with the CUDA toolkit though. I might have a look at improving the error message.

8

I have CUDA 10.1. The same error popped up today after updating my packages. I am on Windows 10 on my side. So no CUPTI either. Updating to 10.2 did not solve the problem, I uninstalled the previous version then I did the complete installation of the new one.

9

I have the same issue using windows 10 and CUDA 10.2.

I have cupti64_102.dll in the directory C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\extras\CUPTI\lib64

It seems like CUDAnative does not try that path:

┌ Debug: Looking for library cupti64, cupti, cupti64_10, cupti64_102
│   locations =
│    3-element Array{String,1}:
│     "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI"
│     "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI\\bin"
│     "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI\\bin\\x64"
└ @ CUDAapi E:\Programs\julia\.julia\packages\CUDAapi\3JKC9\src\CUDAapi.jl:8
[ Info: CUDAnative.jl failed to initialize, GPU functionality unavailable (set JULIA_CUDA_SILENT or JULIA_CUDA_VERBOSE to silence or expand this message)

It appears as if I can work around the issue by manually loading cupti64_102:

julia> using Libdl

julia> ldir = "C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.2\\extras\\CUPTI\\lib64";

julia> Libdl.dlopen(joinpath(ldir, "cupti64_102"), Libdl.RTLD_LAZY;throw_error=false)
Ptr{Nothing} @0x00007ffb79bd0000

julia> using CuArrays

julia> cu([1,2,3])
3-element CuArray{Float32,1,Nothing}:
 1.0
 2.0
 3.0
10

Thanks for the details, I’ll make CUPTI optional for now and also add those search paths to CUDAapi.

11

Should be fixed: CUDAnative failed to initialize, no CUPTI provided · Issue #549 · JuliaGPU/CUDAnative.jl · GitHub

12

In Ubuntu 18.04 you can install CUPTI by:

sudo apt install  libcupti-dev libcupti-doc libcupti9.1