CUDNN in Julia

Is there any way - or what is the best way - to use CUDNN in Julia?

Details:
I am interested in that topic because I want do add GPU-acceleration (Nvidia) to my own little Deep Learning module (it is not available anywere yet). I found out that CUDNN is a common choice for GPU-accelaration for neuronal networks and now I would like to find more about that. I know that it is like a part of CUDA and that it was originally written for C/C++.

Is CUDNN accesibal with using CUDA.jl or is there a good wrapper of CUDNN to Julia?

Best Regards,

Jonas S.

Should be, yes. Try using CUDA.CUDNN.

julia> using CUDA.CUDNN

julia> CUDNN.cudnn

cudnnActivationBackward
cudnnActivationDescriptor
cudnnActivationDescriptorCache
cudnnActivationDescriptorCacheLock
cudnnActivationDescriptor_t
cudnnActivationForward
cudnnActivationForward!
cudnnActivationForwardAD
cudnnActivationForwardWithDefaults
cudnnActivationMode_t
cudnnActivationStruct
cudnnAddTensor
cudnnAddTensor!
cudnnAdvInferVersionCheck
cudnnAdvTrainVersionCheck
cudnnAlgorithmDescriptor_t
cudnnAlgorithmPerformanceStruct
cudnnAlgorithmPerformance_t
cudnnAlgorithmStruct
cudnnAlgorithm_t
cudnnAttnDescriptor
cudnnAttnDescriptorCache
cudnnAttnDescriptorCacheLock
cudnnAttnDescriptor_t
cudnnAttnOutput
cudnnAttnQueryMap_t
cudnnAttnStruct
cudnnBatchNormMode_t
cudnnBatchNormOps_t
cudnnBatchNormalizationBackward
cudnnBatchNormalizationBackwardEx
cudnnBatchNormalizationForwardInference
cudnnBatchNormalizationForwardTraining
cudnnBatchNormalizationForwardTrainingEx
[...]

(Subdirectory of the CUDA.jl repo: CUDA.jl/lib/cudnn at v3.12.0 · JuliaGPU/CUDA.jl · GitHub)

2 Likes

Thank you! But I try to find good tutorials or documentation about CUDNN but I had no succes yet.
I think there are no articles about CUDNN directly in Julia. However, I’m sure there is helpful information about using CUDNN in general.
Do you have any helpful information about this?

There’s several levels of abstraction at which you can use CUDNN:

  • at the lowest level, there’s just the CUDNN C API functions, all of which you can use and are part of the CUDA.CUDNN submodule
  • the same module also has slightly higher-level wrappers (bit more idiomatic, but still true to the CUDNN API). Those sadly aren’t really documented, but there’s a design document by @denizyuret who wrote these wrappers: CUDA.jl/lib/cudnn at master · JuliaGPU/CUDA.jl · GitHub
  • you can also use NNlib, which provides a common API of high-level neural network primitives. There’s a CUDA back-end for these that uses CUDNN, NNlibCUDA.jl
3 Likes

Thanks! I think I will take a look at all of these options.
So if I am using the CUDNN low-level functions provided by CUDA.jl.CUDNN should I use the original docs from Nvidia? NVIDIA Deep Learning cuDNN Documentation

By the way, I am brand new to this forum and I am amazed that (my) questions were answered so quickly.

Yes. The Julia functions generally are just wrappers around the C functions.

Glad to hear that. We try our best :slight_smile:

1 Like