CUDA artifact selection for Pascal GPU

I am using CUDA.jl on a Pascal (sm_61) hardware.

Till now having driver 570xx (575xx also) worked when restricting cuDNN to v1.4.4 (so nvidia cudnn is <v9.10.0, later Pascal is deprecated). This restricted also CUDA.jl to version 5.8.5. Tying runtime_version to v12.x downloaded the right artifact and it worked.

Now I tried to use the latest driver version to support Pascal, 580.xx, and did the same restrictions as above. This time it crashed already using a minimal example with CuArray. dmesg showed me a segfault in libcuda.so.595.58.03 which points to a driver/artifact-library mismatch.

So my question: is it possible to get an artifact having v580 so this mismatch cannot happen?

julia> CUDA.versioninfo()
CUDA toolchain:

  • runtime 12.4, artifact installation
  • driver 580.159.4 for 13.2
  • compiler 12.9

CUDA libraries:

  • CUBLAS: 12.4.5
  • CURAND: 10.3.5
  • CUFFT: 11.2.1
  • CUSOLVER: 11.6.1
  • CUSPARSE: 12.3.1
  • CUPTI: 2024.1.1 (API 12.4.0)
  • NVML: 13.0.0+580.159.4

Julia packages:

  • CUDA: 5.8.5
  • CUDA_Driver_jll: 13.2.1+0
  • CUDA_Compiler_jll: 0.2.2+0
  • CUDA_Runtime_jll: 0.19.2+0

Toolchain:

  • Julia: 1.12.6
  • LLVM: 18.1.7

Preferences:

  • CUDA_Runtime_jll.version: 12.4

1 device:
0: Quadro P2000 (sm_61, 3.682 GiB / 5.000 GiB available)

That is probably a bug in the CUDA forwards compatible driver. Try running with JULIA_CUDA_USE_COMPAT=false.

Hi, I seem to run into a similar issue with a Pascal P1000 GPU.
I am trying to use CUDA in combination with WaterLily but get the following error:

CUDA error: operation not supported (code 801, ERROR_NOT_SUPPORTED)

And the error gets thrown by this line of code:

GPUsim = sphere(3*2^5,2^6;T=Float32,mem=CuArray);

I have fresh installed the latest driver, but this does not fix the issue.

As suggested in this thead I used : JULIA_CUDA_USE_COMPAT: false
But this did not make it work.

I have also tried to use an earlier version of CUDA, version 12.8, by using:
CUDA.set_runtime_version!(v"12.8")

This also has not resolved the issue.

I have the following version info, What strikes me is that my driver is ‘unknown’. What could be my issue?

julia> CUDA.versioninfo()
CUDA toolchain:

  • runtime 12.8.0, artifact installation
  • unknown driver for 13.0
  • compiler 12.9.41, artifact installation

CUDA libraries:

  • cuBLAS: 12.8.4
  • cuSPARSE: 12.5.8
  • cuSOLVER: 11.7.3
  • cuFFT: 11.3.3
  • cuRAND: 10.3.9
  • CUPTI: 2025.1.1 (API 12.8.0)
  • NVML: missing

Julia packages:

  • CUDACore: 6.2.1
  • GPUArrays: 11.5.8
  • GPUCompiler: 1.23.0
  • KernelAbstractions: 0.9.42
  • CUDA_Driver_jll: 13.3.0+1
  • CUDA_Compiler_jll: 0.4.4+1
  • CUDA_Runtime_jll: 0.23.0+1
  • NVPTX_LLVM_Backend_jll: 22.1.7+1

Toolchain:

  • Julia: 1.12.6
  • LLVM: 18.1.7

Environment:

  • JULIA_CUDA_USE_COMPAT: false

Preferences:

  • CUDA_Runtime_jll.version: 12.8

1 device:
0: Quadro P1000 (sm_61, 3.285 GiB / 4.000 GiB available)
compiles to sm_61 / PTX 8.8

Please file an issue including a backtrace and MWE. Or check NVIDIA GeForce GTX 1080 GPU (compute capability 6.1) is not fully supported by CUDA 12.9.0. · Issue #3187 · JuliaGPU/CUDA.jl · GitHub if the issue matches.

FWIW, the JULIA_CUDA_USE_COMPAT workaround should never be needed. The JLL should detect problematic configurations now, as of CUDA_Driver: gate compat driver on device capability - Pull Request #14178 - JuliaPackaging/Yggdrasil - GitHub

Hi,
I have now tried to test CUDA on my desktop PC. This uses a GTX 1060 GPU (instead of Quadro P1000) but still CUDA compute capability 6.1

I have made a MWE where I run two basic CUDA tests and pass them. But the third test fails at the same line as before and gives the same code 801 error:

using CUDA
CUDA.versioninfo()

@assert CUDA.functional()


#simple CUDA test (RESULT: TEST PASSED)
using Test
N = 2^20
x_d = CUDA.fill(1.0f0, N)  # a vector stored on the GPU filled with 1.0 (Float32)
y_d = CUDA.fill(2.0f0, N)  # a vector stored on the GPU filled with 2.0

y_d .+= x_d
@test all(Array(y_d) .== 3.0f0)

#GPU kernel test (RESULT: TEST PASSED)
function gpu_add1!(y, x)
    for i = 1:length(y)
        @inbounds y[i] += x[i]
    end
    return nothing
end

fill!(y_d, 2)
@cuda gpu_add1!(y_d, x_d)
@test all(Array(y_d) .== 3.0f0)

#Test WaterLily package in combination with CUDA
using WaterLily

function sphere(n, m; Re=100, U=1, T=Float64, mem=Array)
    radius, center = m/8, m/2 - 1
    body = AutoBody((x, t) -> √sum(abs2, x .- center) - radius)
    Simulation((n, m, m), (U, 0, 0), 2radius; ν=U*2radius/Re, body, T, mem)
end

GPUsim = sphere(3*2^5, 2^6; T=Float32, mem=CuArray)   # <-- error thrown here

The error gives the following stacktrace:

ERROR: CUDA error: operation not supported (code 801, ERROR_NOT_SUPPORTED)
Stacktrace:
  [1] checked_cuModuleLoadDataEx(_module::Base.RefValue{…}, image::Ptr{…}, numOptions::Int64, options::Vector{…}, optionValues::Vector{…})
    @ CUDACore C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\lib\cudadrv\module.jl:27
  [2] CuModule(data::Vector{UInt8}, options::Dict{CUDACore.CUjit_option_enum, Any})
    @ CUDACore C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\lib\cudadrv\module.jl:59
  [3] CuModule
    @ C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\lib\cudadrv\module.jl:48 [inlined]
  [4] link(job::GPUCompiler.CompilerJob, compiled::@NamedTuple{image::Vector{UInt8}, entry::String})
    @ CUDACore C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\compiler\compilation.jl:504
  [5] actual_compilation(cache::Dict{…}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{…}, compiler::typeof(CUDACore.compile), linker::typeof(CUDACore.link))
    @ GPUCompiler C:\Users\Gebruiker\.julia\packages\GPUCompiler\BSi1T\src\execution.jl:270
  [6] cached_compilation(cache::Dict{…}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{…}, compiler::Function, linker::Function)
    @ GPUCompiler C:\Users\Gebruiker\.julia\packages\GPUCompiler\BSi1T\src\execution.jl:159
  [7] macro expansion
    @ C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\compiler\execution.jl:456 [inlined]
  [8] macro expansion
    @ .\lock.jl:376 [inlined]
  [9] cufunction(f::WaterLily.var"#gpu_##kern_#546#applyV!##1"{…}, tt::Type{…}; kwargs::@Kwargs{…})
    @ CUDACore C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\compiler\execution.jl:451
 [10] cufunction
    @ C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\compiler\execution.jl:448 [inlined]
 [11] #kernel_compile#753
    @ C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\compiler\execution.jl:60 [inlined]
 [12] macro expansion
    @ C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\compiler\execution.jl:183 [inlined]
 [13] (::KernelAbstractions.Kernel{…})(::CuArray{…}, ::Vararg{…}; ndrange::NTuple{…}, workgroupsize::Nothing)
    @ CUDACore.CUDAKernels C:\Users\Gebruiker\.julia\packages\CUDACore\NlVPI\src\CUDAKernels.jl:125
 [14] (::WaterLily.var"##kern#545#applyV!##4")(c::CuArray{Float32, 4, CUDACore.DeviceMemory})
    @ WaterLily C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\core.jl:142
 [15] macro expansion
    @ C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\core.jl:144 [inlined]
 [16] applyV!
    @ C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\Flow.jl:82 [inlined]
 [17] apply!(f::Function, c::CuArray{Float32, 4, CUDACore.DeviceMemory})
    @ WaterLily C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\Flow.jl:81
 [18] Flow(N::Tuple{…}, uBC::Tuple{…}; mem::Type, Δt::Float64, ν::Float64, g::Nothing, u0::Nothing, uλ::Nothing, perdir::Tuple{}, exitBC::Bool, λ::typeof(quick), T::Type)
    @ WaterLily C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\Flow.jl:140
 [19] Flow
    @ C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\Flow.jl:133 [inlined]
 [20] #242
    @ C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\WaterLily.jl:96 [inlined]
 [21] Simulation(dims::Tuple{…}, uBC::Tuple{…}, L::Float64; Δt::Float64, ν::Float64, g::Nothing, U::Nothing, ϵ::Int64, perdir::Tuple{}, u0::Nothing, uλ::Nothing, exitBC::Bool, λ::typeof(quick), body::AutoBody{…}, flow_ctor::WaterLily.var"#240#241"{…}, pois_ctor::WaterLily.var"#244#245"{…}, T::Type, mem::Type)
    @ WaterLily C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\WaterLily.jl:103
 [22] Simulation
    @ C:\Users\Gebruiker\.julia\packages\WaterLily\yOkji\src\WaterLily.jl:93 [inlined]
 [23] #sphere#19
    @ d:\Julia_VisualStudio\Error801_Reproduce_Folder\Error_801.jl:34 [inlined]
 [24] top-level scope
    @ d:\Julia_VisualStudio\Error801_Reproduce_Folder\Error_801.jl:37
Some type information was truncated. Use `show(err)` to see complete types.

I have the following info on CUDA and Julia:

CUDA toolchain: 
- runtime 12.9.0, artifact installation
- unknown driver for 13.0
- compiler 12.9.41, artifact installation

CUDA libraries: 
- cuBLAS: 12.9.1
- cuSPARSE: 12.5.10
- cuSOLVER: 11.7.5
- cuFFT: 11.4.1
- cuRAND: 10.3.10
- CUPTI: 2025.2.1 (API 12.9.1)
- NVML: missing

Julia packages: 
- CUDACore: 6.2.1
- GPUArrays: 11.5.8
- GPUCompiler: 1.23.0
- KernelAbstractions: 0.9.42
- CUDA_Driver_jll: 13.3.0+1
- CUDA_Compiler_jll: 0.4.4+1
- CUDA_Runtime_jll: 0.23.0+1
- NVPTX_LLVM_Backend_jll: 22.1.7+1

Toolchain:
- Julia: 1.12.6
- LLVM: 18.1.7

1 device:
  0: NVIDIA GeForce GTX 1060 6GB (sm_61, 5.075 GiB / 6.000 GiB available)
     compiles to sm_61 / PTX 8.8

Please refer to NVIDIA GeForce GTX 1080 GPU (compute capability 6.1) is not fully supported by CUDA 12.9.0. · Issue #3187 · JuliaGPU/CUDA.jl · GitHub