Could not download forward compatibility package for CUDA

Pavel_Petrov · December 26, 2022, 12:03pm

I am trying to run some CUDA calculations on a cluster and I have a problem with CUDA artifacts. Here is a simple test code, which I run on cluster node.

ENV["JULIA_CUDA_USE_BINARYBUILDER"] = false
using CUDA
CUDA.versioninfo()
println("")
a = CuArray([1,2])
print(a)

The code give me the output, which seems to be correct:

CUDA toolkit 11.6, local installation
NVIDIA driver 460.32.3, for CUDA 11.2
CUDA driver 11.2

Libraries: 
- CUBLAS: 11.9.2
- CURAND: 10.2.9
- CUFFT: 10.7.2
- CUSOLVER: 11.3.4
- CUSPARSE: 11.7.2
- CUPTI: 16.0.0
- NVML: 11.0.0+460.32.3
- CUDNN: 8.10.1 (for CUDA 11.2.0)
- CUTENSOR: missing

Toolchain:
- Julia: 1.6.7
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

Environment:
- JULIA_CUDA_USE_BINARYBUILDER: false

2 devices:
  0: Tesla K40m (sm_35, 11.918 GiB / 11.921 GiB available)
  1: Tesla K40m (sm_35, 11.918 GiB / 11.921 GiB available)

[1, 2]

But in addition I have some error messages in the debug output. Is this OK? What is wrong
with CUDA_compat? Can I get rid of this “Downloading artifact: CUDA_compat” messages?

 Downloading artifact: CUDA_compat
 Downloading artifact: CUDA_compat
┌ Debug: Could not download forward compatibility package
│   exception =
│    Unable to automatically install 'CUDA_compat' from '/home/.julia/packages/CUDA/DfvRa/Artifacts.toml'
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:33
│      [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.PipeEndpoint)
│        @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:446
│      [3] ensure_artifact_installed(name::String, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, verbose::Bool, quiet_download::Bool, io::Base.PipeEndpoint)
│        @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:409
│      [4] _artifact_str(__module__::Module, artifacts_toml::String, name::SubString{String}, path_tail::String, artifact_dict::Dict{String, Any}, hash::Base.SHA1, platform::Base.BinaryPlatforms.Platform, lazyartifacts::Any)
│        @ Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:548
│      [5] #invokelatest#2
│        @ ./essentials.jl:708 [inlined]
│      [6] invokelatest
│        @ ./essentials.jl:706 [inlined]
│      [7] macro expansion
│        @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:690 [inlined]
│      [8] (::CUDA.var"#1#4")(id::String)
│        @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:97
│      [9] find_libcuda()
│        @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:98
│     [10] libcuda()
│        @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:141
│     [11] macro expansion
│        @ ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:67 [inlined]
│     [12] macro expansion
│        @ lock.jl:209 [inlined]
│     [13] toolkit()
│        @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:21
│     [14] macro expansion
│        @ ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:342 [inlined]
│     [15] macro expansion
│        @ lock.jl:209 [inlined]
│     [16] libcudart()
│        @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:21
│     [17] macro expansion
│        @ ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/error.jl:95 [inlined]
│     [18] runtime_version()
│        @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/version.jl:44
│     [19] versioninfo(io::Base.PipeEndpoint) (repeats 2 times)
│        @ CUDA ~/.julia/packages/CUDA/DfvRa/src/utilities.jl:32
│     [20] top-level scope
│        @ ~/julia/Test_CUDA/test_CUDA.jl:5
│     [21] include(mod::Module, _path::String)
│        @ Base ./Base.jl:384
│     [22] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:285
│     [23] _start()
│        @ Base ./client.jl:485
└ @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:100
┌ Debug: Trying to use local installation...
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:210
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_PATH, CUDA_HOME
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:271
┌ Debug: Looking for binary nvdisasm in /opt/software/nvidia/cuda-11.6u2
│   all_locations =
│    2-element Vector{String}:
│     "/opt/software/nvidia/cuda-11.6u2"
│     "/opt/software/nvidia/cuda-11.6u2/bin"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:147
┌ Debug: Found nvdisasm at /opt/software/nvidia/cuda-11.6u2/bin/nvdisasm
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:153
┌ Debug: Looking for library cudart, no specific version, in /opt/software/nvidia/cuda-11.6u2
│   all_names =
│    1-element Vector{String}:
│     "libcudart.so"
│   all_locations =
│    4-element Vector{String}:
│     "/opt/software/nvidia/cuda-11.6u2"
│     "/opt/software/nvidia/cuda-11.6u2/lib"
│     "/opt/software/nvidia/cuda-11.6u2/lib64"
│     "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcudart.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: Found local CUDA 11.6.0 at /opt/software/nvidia/cuda-11.6u2
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:232
┌ Debug: Looking for library cudart, no specific version, in /opt/software/nvidia/cuda-11.6u2
│   all_names =
│    1-element Vector{String}:
│     "libcudart.so"
│   all_locations =
│    4-element Vector{String}:
│     "/opt/software/nvidia/cuda-11.6u2"
│     "/opt/software/nvidia/cuda-11.6u2/lib"
│     "/opt/software/nvidia/cuda-11.6u2/lib64"
│     "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcudart.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: Looking for library cublasLt, no specific version, in /opt/software/nvidia/cuda-11.6u2
│   all_names =
│    1-element Vector{String}:
│     "libcublasLt.so"
│   all_locations =
│    4-element Vector{String}:
│     "/opt/software/nvidia/cuda-11.6u2"
│     "/opt/software/nvidia/cuda-11.6u2/lib"
│     "/opt/software/nvidia/cuda-11.6u2/lib64"
│     "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcublasLt.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: Looking for library cublas, no specific version, in /opt/software/nvidia/cuda-11.6u2
│   all_names =
│    1-element Vector{String}:
│     "libcublas.so"
│   all_locations =
│    4-element Vector{String}:
│     "/opt/software/nvidia/cuda-11.6u2"
│     "/opt/software/nvidia/cuda-11.6u2/lib"
│     "/opt/software/nvidia/cuda-11.6u2/lib64"
│     "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcublas.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=0
│   value: type=int; val=POINTER (IN HEX:0x0x2b688951add0)
│  Time: 2022-12-26T14:44:46 elapsed from start 0.016667 minutes or 1.000000 seconds
│ Process=150087; Thread=47727540146624; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=1
│   value: type=int; val=POINTER (IN HEX:0x0x2b6889c20c10)
│  Time: 2022-12-26T14:44:46 elapsed from start 0.016667 minutes or 1.000000 seconds
│ Process=150087; Thread=47727540146624; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
┌ Debug:  cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│   type: type=SOME TYPE; val=2
│   value: type=int; val=POINTER (IN HEX:0x0x2b6889c20c20)
│  Time: 2022-12-26T14:44:46 elapsed from start 0.016667 minutes or 1.000000 seconds
│ Process=150087; Thread=47727540146624; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│  COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
│

Ahmed_Salih · December 26, 2022, 12:20pm

I am not too much into details, but I noticed this.

By having this enabled, I think it will want you to install/direct to your own CUDA artifact?

This is atleast what I get from reading the docs on CUDA

Kind regards

Pavel_Petrov · December 26, 2022, 12:36pm

When you work with cluster nodes, usually you have to use CUDA drivers which already installed on the cluster. I have loaded it as modules.

module load nvidia/cuda/11.6u2
module load nvidia/cudnn/8.1.1

As far as I know from docs and forum messages in this case you have to switch off a JULIA_CUDA_USE_BINARYBUILDER and let to use locally built binaries. I can remove this line, but it will try to download all the CUDA packages, not only CUDA_compat.

Pavel_Petrov · December 28, 2022, 8:43am

Just if anybody need it: I have found a solution.
In file “~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl” I found an “IF” operator which control the CUDA_compat downloading. Obviously, my CUDA toolkit 11.6 is a bit too old and CUDAdrv.jl try to download compatibility package.

    if getenv("JULIA_CUDA_USE_COMPAT", !hooked && !system_driver_loaded) && _system_version[] < v"11.7"

Solved it easily just switching off JULIA_CUDA_USE_COMPAT environment variable

export JULIA_CUDA_USE_BINARYBUILDER=false
export JULIA_CUDA_USE_COMPAT=false

maleadt · December 28, 2022, 10:32am

Why would you disable the attempt to load the forward-compatibility driver? The debug message mentioned that your system isn’t compatible with it, nothing more. It’s just a debug message.

Pavel_Petrov · December 28, 2022, 11:09am

User cluster nodes have GPU but have no an internet access. I can install julia packages using a login node but I have no possibility to download when I run code on cluster nodes with GPU.
As far as I found out in message boards - It is not so rare situation

In my case, an unsuccessful attempt to download CUDA_compat takes about 20 additional seconds of compilation time. Thus, I prefer to switch it off. I think that correct solution is just to ask the cluster admin to update CUDA driver, but it could take a while

maleadt · January 2, 2023, 11:40am

That would be very surprising. Are you sure you’re not confounding the failed download of CUDA_Compat with the artifacts? The latter tries a bunch of time, for each CUDA version, so might take a while to fail all the way. Or is this a feature of your environment (e.g., by timing out connections instead of rejecting them)?

Pavel_Petrov · January 5, 2023, 8:55am

I believe that your last suggestion is correct. I have tried to start wget on the node with GPU. Here is the log and there was the very same delay - approximately 20 seconds.

$ wget https://google.com
--2023-01-05 11:51:18--  https://google.com/
Resolving google.com (google.com)... 108.177.14.101, 108.177.14.138, 108.177.14.139, ...
Connecting to google.com (google.com)|108.177.14.101|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.138|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.139|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.102|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.113|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.100|:443... failed: No route to host.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::66|:443... failed: Network is unreachable.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::8a|:443... failed: Network is unreachable.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::8b|:443... failed: Network is unreachable.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::65|:443... failed: Network is unreachable.