[CUDA] Issue with cached_memory() and Quadro RTX

I have a new toy to play with (a Quadro RTX) and I want to use CUDA.jl with the GPU.

I get a quite simple yet annoying issue associated to the function cached_memory.
The very simple MWE


julia> CUDA.versioninfo()
CUDA toolkit 11.4.1, artifact installation
CUDA driver 11.2.0
NVIDIA driver 460.91.3

Libraries: 
- CUBLAS: 11.5.4
- CURAND: 10.2.5
- CUFFT: 10.5.1
- CUSOLVER: 11.2.0
- CUSPARSE: 11.6.0
- CUPTI: 14.0.0
- NVML: 11.0.0+460.91.3
- CUDNN: 8.20.2 (for CUDA 11.4.0)
- CUTENSOR: 1.3.0 (for CUDA 11.2.0)

Toolchain:
- Julia: 1.6.2
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

1 device:
  0: Quadro RTX 5000 (sm_75, 14.922 GiB / 15.738 GiB available)

julia> CUDA.cached_memory()
ERROR: CUDA error: invalid argument (code 1, ERROR_INVALID_VALUE)
Stacktrace:
 [1] throw_api_error(res::CUDA.cudaError_enum)
   @ CUDA ~/.julia/packages/CUDA/VGl9W/lib/cudadrv/error.jl:105
 [2] macro expansion
   @ ~/.julia/packages/CUDA/VGl9W/lib/cudadrv/error.jl:115 [inlined]
 [3] cuMemPoolGetAttribute(pool::CuMemoryPool, attr::CUDA.CUmemPool_attribute_enum, value::Base.RefValue{UInt64})
   @ CUDA ~/.julia/packages/CUDA/VGl9W/lib/utils/call.jl:26
 [4] attribute(X::Type, pool::CuMemoryPool, attr::CUDA.CUmemPool_attribute_enum)
   @ CUDA ~/.julia/packages/CUDA/VGl9W/lib/cudadrv/pool.jl:77
 [5] cached_memory()
   @ CUDA ~/.julia/packages/CUDA/VGl9W/src/pool.jl:532
 [6] top-level scope
   @ REPL[8]:100: 

The main issue is that the function cached_memory() is used several times (for instance for allocation) at several places (like in cudnnFindConvolutionAlgorithmWorkspaceSize for conv in GPU)

Any hints to address that ?

Already fixed, Fix cached_memory on CUDA 11.2. · JuliaGPU/CUDA.jl@e0d5b69 · GitHub, I’ll create a backport release.

2 Likes

Wooo, that was fast :smiley:
Thanks a lot !

New version has been tagged: New version: CUDA v3.4.2 by JuliaRegistrator · Pull Request #43682 · JuliaRegistries/General · GitHub

2 Likes