CUDNN in Julia

Specific Domains GPU
1

Is there any way - or what is the best way - to use CUDNN in Julia?

Details:
I am interested in that topic because I want do add GPU-acceleration (Nvidia) to my own little Deep Learning module (it is not available anywere yet). I found out that CUDNN is a common choice for GPU-accelaration for neuronal networks and now I would like to find more about that. I know that it is like a part of CUDA and that it was originally written for C/C++.

Is CUDNN accesibal with using CUDA.jl or is there a good wrapper of CUDNN to Julia?

Best Regards,

Jonas S.

2

Should be, yes. Try using CUDA.CUDNN.

julia> using CUDA.CUDNN

julia> CUDNN.cudnn

cudnnActivationBackward
cudnnActivationDescriptor
cudnnActivationDescriptorCache
cudnnActivationDescriptorCacheLock
cudnnActivationDescriptor_t
cudnnActivationForward
cudnnActivationForward!
cudnnActivationForwardAD
cudnnActivationForwardWithDefaults
cudnnActivationMode_t
cudnnActivationStruct
cudnnAddTensor
cudnnAddTensor!
cudnnAdvInferVersionCheck
cudnnAdvTrainVersionCheck
cudnnAlgorithmDescriptor_t
cudnnAlgorithmPerformanceStruct
cudnnAlgorithmPerformance_t
cudnnAlgorithmStruct
cudnnAlgorithm_t
cudnnAttnDescriptor
cudnnAttnDescriptorCache
cudnnAttnDescriptorCacheLock
cudnnAttnDescriptor_t
cudnnAttnOutput
cudnnAttnQueryMap_t
cudnnAttnStruct
cudnnBatchNormMode_t
cudnnBatchNormOps_t
cudnnBatchNormalizationBackward
cudnnBatchNormalizationBackwardEx
cudnnBatchNormalizationForwardInference
cudnnBatchNormalizationForwardTraining
cudnnBatchNormalizationForwardTrainingEx
[...]

(Subdirectory of the CUDA.jl repo: CUDA.jl/lib/cudnn at v3.12.0 · JuliaGPU/CUDA.jl · GitHub)

2 Likes
4

Thank you! But I try to find good tutorials or documentation about CUDNN but I had no succes yet.
I think there are no articles about CUDNN directly in Julia. However, I’m sure there is helpful information about using CUDNN in general.
Do you have any helpful information about this?

5

There’s several levels of abstraction at which you can use CUDNN:

  • at the lowest level, there’s just the CUDNN C API functions, all of which you can use and are part of the CUDA.CUDNN submodule
  • the same module also has slightly higher-level wrappers (bit more idiomatic, but still true to the CUDNN API). Those sadly aren’t really documented, but there’s a design document by @denizyuret who wrote these wrappers: CUDA.jl/lib/cudnn at master · JuliaGPU/CUDA.jl · GitHub
  • you can also use NNlib, which provides a common API of high-level neural network primitives. There’s a CUDA back-end for these that uses CUDNN, NNlibCUDA.jl
3 Likes
7

Thanks! I think I will take a look at all of these options.
So if I am using the CUDNN low-level functions provided by CUDA.jl.CUDNN should I use the original docs from Nvidia? NVIDIA Deep Learning cuDNN Documentation

By the way, I am brand new to this forum and I am amazed that (my) questions were answered so quickly.

8

Yes. The Julia functions generally are just wrappers around the C functions.

Glad to hear that. We try our best :slight_smile:

1 Like