CUDNN produces a lot of errors and warnings

Euhan · February 2, 2023, 4:42pm

I have lately been getting a lot of errors emanating from CUDNN.jl when training Flux models. This happens also with code I really think has or would have worked before.

Does anyone understand this?

┌ Error: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 85 message(s)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: ptr.isSupported()
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: engine_post_checks(handle, *ebuf.get(), engine.getPerfKnobs(), req_size)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: finalize_internal()
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:35.819594 (0d+0h+2m+0s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 29 message(s)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│ Time: 2023-02-01T11:36:39.049080 (0d+0h+2m+4s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Warning: CuDNN (v8302) function cudnnBatchNormalizationForwardTraining() called:
│     Info: Traceback contains 1 message(s)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: !canRunSemiPersist
│ Time: 2023-02-01T11:36:45.400175 (0d+0h+2m+10s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 85 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.612761 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.613024 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 65 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│ Time: 2023-02-01T11:36:52.643631 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.643796 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnBatchNormalizationForwardTraining() called:
│     Info: Traceback contains 1 message(s)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: !canRunSemiPersist
│ Time: 2023-02-01T11:36:52.643947 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 94 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.689861 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.690034 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 74 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│ Time: 2023-02-01T11:36:52.703894 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.704060 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnBatchNormalizationForwardTraining() called:
│     Info: Traceback contains 1 message(s)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: !canRunSemiPersist
│ Time: 2023-02-01T11:36:52.704167 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 94 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.731834 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.732003 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 74 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│ Time: 2023-02-01T11:36:52.748695 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.748856 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Warning: CuDNN (v8302) function cudnnBatchNormalizationForwardTraining() called:
│     Info: Traceback contains 1 message(s)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: !canRunSemiPersist
│ Time: 2023-02-01T11:36:52.748991 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:138
┌ Error: CuDNN (v8302) function cudnnGetConvolutionForwardAlgorithmMaxCount() called:
│     Info: Traceback contains 94 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: false == cudnn::cnn::isForwardSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Warning: CUDNN_STATUS_NOT_SUPPORTED; Reason: T_ENGINEMAP::isLegacyAlgoSupported(handle, xDesc, wDesc, cDesc, yDesc, algo)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.758726 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
┌ Error: CuDNN (v8302) function cudnnConvolutionForward() called:
│     Info: Traceback contains 3 message(s)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: dimA[i] <= 0
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: cudnn::ops::setTensorNdDescriptor(desc, dtype, nbDims, dimA, strideA, true)
│         Error: CUDNN_STATUS_BAD_PARAM; Reason: initStatus = getXDescriptor(conv, &xDescCompat)
│ Time: 2023-02-01T11:36:52.758896 (0d+0h+2m+17s since start)
│ Process=50017; Thread=50017; GPU=NULL; Handle=NULL; StreamId=NULL.
└ @ CUDA.CUDNN ~/.julia/packages/CUDA/Ey3w2/lib/cudnn/CUDNN.jl:140
⋮

It goes on for another 30 000 lines, but even the output above have been heavily edited to keep up the dramatic tension of the narrative.

Extra info:
Julia: 1.8.5 (running on linux)
CUDA: 3.12.1
Flux: 0.13.11
The underlying CUDA library reported:
Driver version: 12.0.0
System driver version: 12.0.0

Euhan · February 7, 2023, 12:16pm

I think I know now that this behaviour only occurred when I ran julia with the -g 2 option.

That begs the question whether this goes on all the time in working code, just not being reported. In that case I don’t understand the difference between warnings and errors, since none of them appears to be fatal.

maleadt · February 8, 2023, 9:14am

Yeah we only show the debug log when running with -g2 or when JULIA_DEBUG includes cuDNN: CUDA.jl/cuDNN.jl at 0dce8ca9022f9c948dd8a7c7bf2f070d8218db05 · JuliaGPU/CUDA.jl · GitHub

Maybe the error gets handled appropriately by the caller, and thus the debug message isn’t a problem. But it’d be good to ensure that’s the case.