# Could not download forward compatibility package for CUDA

**URL:** <https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142>\
**Category:** General Usage\
**Tags:** cuda\
**Created:** [December 26, 2022, 12:03pm UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142 "2022-12-26T12:03:35Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pavel\_Petrov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_petrov/32/45258_2.png) [@Pavel\_Petrov](https://discourse.julialang.org/u/Pavel_Petrov)\
**Post date:** [December 26, 2022, 12:03pm UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/1 "2022-12-26T12:03:35Z")

</div>

I am trying to run some CUDA calculations on a cluster and I have a problem with CUDA artifacts. Here is a simple test code, which I run on cluster node.

```julia
ENV["JULIA_CUDA_USE_BINARYBUILDER"] = false
using CUDA
CUDA.versioninfo()
println("")
a = CuArray([1,2])
print(a)

```

The code give me the output, which seems to be correct:

```julia
CUDA toolkit 11.6, local installation
NVIDIA driver 460.32.3, for CUDA 11.2
CUDA driver 11.2

Libraries: 
- CUBLAS: 11.9.2
- CURAND: 10.2.9
- CUFFT: 10.7.2
- CUSOLVER: 11.3.4
- CUSPARSE: 11.7.2
- CUPTI: 16.0.0
- NVML: 11.0.0+460.32.3
- CUDNN: 8.10.1 (for CUDA 11.2.0)
- CUTENSOR: missing

Toolchain:
- Julia: 1.6.7
- LLVM: 11.0.1
- PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0
- Device capability support: sm_35, sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80

Environment:
- JULIA_CUDA_USE_BINARYBUILDER: false

2 devices:
  0: Tesla K40m (sm_35, 11.918 GiB / 11.921 GiB available)
  1: Tesla K40m (sm_35, 11.918 GiB / 11.921 GiB available)

[1, 2]

```

But in addition I have some error messages in the debug output. Is this OK? What is wrong  
with CUDA\_compat? Can I get rid of this “Downloading artifact: CUDA\_compat” messages?

```julia
 Downloading artifact: CUDA_compat
 Downloading artifact: CUDA_compat
┌ Debug: Could not download forward compatibility package
│ exception =
│ Unable to automatically install 'CUDA_compat' from '/home/.julia/packages/CUDA/DfvRa/Artifacts.toml'
│ Stacktrace:
│ [1] error(s::String)
│ @ Base ./error.jl:33
│ [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::Base.PipeEndpoint)
│ @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:446
│ [3] ensure_artifact_installed(name::String, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, pkg_uuid::Nothing, verbose::Bool, quiet_download::Bool, io::Base.PipeEndpoint)
│ @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Artifacts.jl:409
│ [4] _artifact_str( __module__ ::Module, artifacts_toml::String, name::SubString{String}, path_tail::String, artifact_dict::Dict{String, Any}, hash::Base.SHA1, platform::Base.BinaryPlatforms.Platform, lazyartifacts::Any)
│ @ Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:548
│ [5] #invokelatest#2
│ @ ./essentials.jl:708 [inlined]
│ [6] invokelatest
│ @ ./essentials.jl:706 [inlined]
│ [7] macro expansion
│ @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Artifacts/src/Artifacts.jl:690 [inlined]
│ [8] (::CUDA.var"#1#4")(id::String)
│ @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:97
│ [9] find_libcuda()
│ @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:98
│ [10] libcuda()
│ @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:141
│ [11] macro expansion
│ @ ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:67 [inlined]
│ [12] macro expansion
│ @ lock.jl:209 [inlined]
│ [13] toolkit()
│ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:21
│ [14] macro expansion
│ @ ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:342 [inlined]
│ [15] macro expansion
│ @ lock.jl:209 [inlined]
│ [16] libcudart()
│ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:21
│ [17] macro expansion
│ @ ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/error.jl:95 [inlined]
│ [18] runtime_version()
│ @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/version.jl:44
│ [19] versioninfo(io::Base.PipeEndpoint) (repeats 2 times)
│ @ CUDA ~/.julia/packages/CUDA/DfvRa/src/utilities.jl:32
│ [20] top-level scope
│ @ ~/julia/Test_CUDA/test_CUDA.jl:5
│ [21] include(mod::Module, _path::String)
│ @ Base ./Base.jl:384
│ [22] exec_options(opts::Base.JLOptions)
│ @ Base ./client.jl:285
│ [23] _start()
│ @ Base ./client.jl:485
└ @ CUDA ~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl:100
┌ Debug: Trying to use local installation...
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:210
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_PATH, CUDA_HOME
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:271
┌ Debug: Looking for binary nvdisasm in /opt/software/nvidia/cuda-11.6u2
│ all_locations =
│ 2-element Vector{String}:
│ "/opt/software/nvidia/cuda-11.6u2"
│ "/opt/software/nvidia/cuda-11.6u2/bin"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:147
┌ Debug: Found nvdisasm at /opt/software/nvidia/cuda-11.6u2/bin/nvdisasm
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:153
┌ Debug: Looking for library cudart, no specific version, in /opt/software/nvidia/cuda-11.6u2
│ all_names =
│ 1-element Vector{String}:
│ "libcudart.so"
│ all_locations =
│ 4-element Vector{String}:
│ "/opt/software/nvidia/cuda-11.6u2"
│ "/opt/software/nvidia/cuda-11.6u2/lib"
│ "/opt/software/nvidia/cuda-11.6u2/lib64"
│ "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcudart.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: Found local CUDA 11.6.0 at /opt/software/nvidia/cuda-11.6u2
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/bindeps.jl:232
┌ Debug: Looking for library cudart, no specific version, in /opt/software/nvidia/cuda-11.6u2
│ all_names =
│ 1-element Vector{String}:
│ "libcudart.so"
│ all_locations =
│ 4-element Vector{String}:
│ "/opt/software/nvidia/cuda-11.6u2"
│ "/opt/software/nvidia/cuda-11.6u2/lib"
│ "/opt/software/nvidia/cuda-11.6u2/lib64"
│ "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcudart.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: Looking for library cublasLt, no specific version, in /opt/software/nvidia/cuda-11.6u2
│ all_names =
│ 1-element Vector{String}:
│ "libcublasLt.so"
│ all_locations =
│ 4-element Vector{String}:
│ "/opt/software/nvidia/cuda-11.6u2"
│ "/opt/software/nvidia/cuda-11.6u2/lib"
│ "/opt/software/nvidia/cuda-11.6u2/lib64"
│ "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcublasLt.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: Looking for library cublas, no specific version, in /opt/software/nvidia/cuda-11.6u2
│ all_names =
│ 1-element Vector{String}:
│ "libcublas.so"
│ all_locations =
│ 4-element Vector{String}:
│ "/opt/software/nvidia/cuda-11.6u2"
│ "/opt/software/nvidia/cuda-11.6u2/lib"
│ "/opt/software/nvidia/cuda-11.6u2/lib64"
│ "/opt/software/nvidia/cuda-11.6u2/libx64"
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:119
┌ Debug: Found libcublas.so at /opt/software/nvidia/cuda-11.6u2/lib64
└ @ CUDA.Deps ~/.julia/packages/CUDA/DfvRa/deps/discovery.jl:128
┌ Debug: cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│ type: type=SOME TYPE; val=0
│ value: type=int; val=POINTER (IN HEX:0x0x2b688951add0)
│ Time: 2022-12-26T14:44:46 elapsed from start 0.016667 minutes or 1.000000 seconds
│ Process=150087; Thread=47727540146624; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
┌ Debug: cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│ type: type=SOME TYPE; val=1
│ value: type=int; val=POINTER (IN HEX:0x0x2b6889c20c10)
│ Time: 2022-12-26T14:44:46 elapsed from start 0.016667 minutes or 1.000000 seconds
│ Process=150087; Thread=47727540146624; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
└ @ CUDA.CUBLAS ~/.julia/packages/CUDA/DfvRa/lib/cublas/CUBLAS.jl:222
┌ Debug: cuBLAS (v11.5) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called:
│ type: type=SOME TYPE; val=2
│ value: type=int; val=POINTER (IN HEX:0x0x2b6889c20c20)
│ Time: 2022-12-26T14:44:46 elapsed from start 0.016667 minutes or 1.000000 seconds
│ Process=150087; Thread=47727540146624; GPU=0; Handle=POINTER (IN HEX:0x(nil))
│ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3)
│ 

```

---

<div class="post-metadata">

**Author:** ![Ahmed\_Salih](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ahmed_salih/32/206579_2.png) [@Ahmed\_Salih](https://discourse.julialang.org/u/Ahmed_Salih)\
**Post date:** [December 26, 2022, 12:20pm UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/2 "2022-12-26T12:20:24Z")

</div>

> [@Pavel\_Petrov](#):
>
> ```julia
> Environment:
> - JULIA_CUDA_USE_BINARYBUILDER: false
> 
> ```

I am not too much into details, but I noticed this.

By having this enabled, I think it will want you to install/direct to your own CUDA artifact?

This is atleast what I get from reading the docs on CUDA

Kind regards

---

<div class="post-metadata">

**Author:** ![Pavel\_Petrov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_petrov/32/45258_2.png) [@Pavel\_Petrov](https://discourse.julialang.org/u/Pavel_Petrov)\
**Post date:** [December 26, 2022, 12:36pm UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/3 "2022-12-26T12:36:43Z")

</div>

When you work with cluster nodes, usually you have to use CUDA drivers which already installed on the cluster. I have loaded it as modules.

```julia
module load nvidia/cuda/11.6u2
module load nvidia/cudnn/8.1.1

```

As far as I know from docs and forum messages in this case you have to switch off a JULIA\_CUDA\_USE\_BINARYBUILDER and let to use locally built binaries. I can remove this line, but it will try to download all the CUDA packages, not only CUDA\_compat.

---

<div class="post-metadata">

**Author:** ![Pavel\_Petrov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_petrov/32/45258_2.png) [@Pavel\_Petrov](https://discourse.julialang.org/u/Pavel_Petrov)\
**Post date:** [December 28, 2022, 8:43am UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/4 "2022-12-28T08:43:14Z")

</div>

Just if anybody need it: I have found a solution.  
In file “~/.julia/packages/CUDA/DfvRa/lib/cudadrv/CUDAdrv.jl” I found an “IF” operator which control the CUDA\_compat downloading. Obviously, my CUDA toolkit 11.6 is a bit too old and CUDAdrv.jl try to download compatibility package.

```julia
    if getenv("JULIA_CUDA_USE_COMPAT", !hooked && !system_driver_loaded) && _system_version[] < v"11.7"

```

Solved it easily just switching off JULIA\_CUDA\_USE\_COMPAT environment variable

```julia
export JULIA_CUDA_USE_BINARYBUILDER=false
export JULIA_CUDA_USE_COMPAT=false

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [December 28, 2022, 10:32am UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/5 "2022-12-28T10:32:36Z")

</div>

Why would you disable the attempt to load the forward-compatibility driver? The debug message mentioned that your system isn’t compatible with it, nothing more. It’s just a debug message.

---

<div class="post-metadata">

**Author:** ![Pavel\_Petrov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_petrov/32/45258_2.png) [@Pavel\_Petrov](https://discourse.julialang.org/u/Pavel_Petrov)\
**Post date:** [December 28, 2022, 11:09am UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/6 "2022-12-28T11:09:53Z")

</div>

User cluster nodes have GPU but have no an internet access. I can install julia packages using a login node but I have no possibility to download when I run code on cluster nodes with GPU.  
As far as I found out in message boards - It is not so [rare situation](https://discourse.julialang.org/t/how-to-use-cuda-on-cluster-nodes-without-internet-access/79190)

In my case, an unsuccessful attempt to download CUDA\_compat takes about 20 additional seconds of compilation time. Thus, I prefer to switch it off. I think that correct solution is just to ask the cluster admin to update CUDA driver, but it could take a while

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 2, 2023, 11:40am UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/7 "2023-01-02T11:40:42Z")

</div>

> [@Pavel\_Petrov](#):
>
> an unsuccessful attempt to download CUDA\_compat takes about 20 additional seconds of compilation time

That would be very surprising. Are you sure you’re not confounding the failed download of CUDA\_Compat with the artifacts? The latter tries a bunch of time, for each CUDA version, so might take a while to fail all the way. Or is this a feature of your environment (e.g., by timing out connections instead of rejecting them)?

---

<div class="post-metadata">

**Author:** ![Pavel\_Petrov](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pavel_petrov/32/45258_2.png) [@Pavel\_Petrov](https://discourse.julialang.org/u/Pavel_Petrov)\
**Post date:** [January 5, 2023, 8:55am UTC](https://discourse.julialang.org/t/could-not-download-forward-compatibility-package-for-cuda/92142/8 "2023-01-05T08:55:05Z")

</div>

I believe that your last suggestion is correct. I have tried to start wget on the node with GPU. Here is the log and there was the very same delay - approximately 20 seconds.

```julia
$ wget https://google.com
--2023-01-05 11:51:18-- https://google.com/
Resolving google.com (google.com)... 108.177.14.101, 108.177.14.138, 108.177.14.139, ...
Connecting to google.com (google.com)|108.177.14.101|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.138|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.139|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.102|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.113|:443... failed: No route to host.
Connecting to google.com (google.com)|108.177.14.100|:443... failed: No route to host.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::66|:443... failed: Network is unreachable.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::8a|:443... failed: Network is unreachable.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::8b|:443... failed: Network is unreachable.
Connecting to google.com (google.com)|2a00:1450:4010:c0f::65|:443... failed: Network is unreachable.

```