CuArrays warning: You are using CUDNN 7.1.4 for CUDA 9.0.0 with CUDA toolkit 9.2.148; these might be incompatible

Specific Domains GPU
1

I was trying to run the Flux modelzoo MNIST mlp.jl example and I notice these warning coming from CuArrays.jl:

julia> include("mlp.jl")
[ Info: Precompiling Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
┌ Warning: CuArrays.jl only supports CUDNN v7.6 or higher
└ @ CuArrays ~/.julia/packages/CuArrays/A6GUx/src/CuArrays.jl:122
┌ Warning: You are using CUDNN 7.1.4 for CUDA 9.0.0 with CUDA toolkit 9.2.148; these might be incompatible.

However, I’ve got CUDNN 7.6.5 installed and nvidia-smi reports that I’m on CUDA Version 10.1.

Why does CuArrays.jl think I’m using CUDA 9.0.0 and CUDNN 7.1.4?
Is there some configuration to tell CuArrays.jl to use different versions of CUDA/CUDNN libs?

I also get the same warnings above if I just do:

pkg> test CuArrays

Julia 1.3.1, CuArrays v1.7.3

2

Did you see my comments on Slack? Copying them here:

maybe you have multiple libcudnn’s on your system?
try running with JULIA_DEBUG=all to see which CUDA and libcudnn it picked up

The CUDA compatibility version of your driver, as reported by nvidia-smi, is separate from the actual CUDA toolkit you have installed.

3

Sorry, didn’t see the response on slack…

I’m pretty sure there are multiple CUDNN libraries installed on my system, do I just get rid of all but the latest or is there a way to tell Julia to use a specific one (via environment variable or some similar mechanism)?

4

There aren’t specific env vars to override cudnn, but if it’s at the same root as a compatible CUDA installation you can define CUDA_PATH to have that override detection. And even easier: the next version of CUDAnative/CuArrays will use its own CUDA/libcudnn through BinaryBuilder.

5

I figured it out. I had an old version of CUDNN installed in a directory and LD_LIBRARY_PATH was pointing to it. Most likely from some old instructions for getting an old version of Tensorflow running.