Hi everyone,
I am new to CUDA.jl and am receiving the following error issues when performing using CUDA on a GPU node of a Linux cluster.
Precompiling CUDA...
22064.9 ms ✓ InvertedIndices
22158.5 ms ✓ LaTeXStrings
22141.4 ms ✓ Reexport
22141.0 ms ✓ InlineStrings
22134.9 ms ✓ UnsafeAtomics
22167.7 ms ✓ AbstractFFTs
22136.1 ms ✓ DataAPI
22144.2 ms ✓ Statistics
22311.0 ms ✓ LLVMLoopInfo
22174.0 ms ✓ DataValueInterfaces
22178.5 ms ✓ OrderedCollections
22330.1 ms ✓ IteratorInterfaceExtensions
22355.6 ms ✓ ExprTools
22347.6 ms ✓ BFloat16s
53222.9 ms ✓ Crayons
33646.0 ms ✓ CEnum
33716.4 ms ✓ StaticArraysCore
34388.7 ms ✓ HashArrayMappedTries
34377.1 ms ✓ Atomix
34370.4 ms ✓ TableTraits
34691.2 ms ✓ PooledArrays
34816.2 ms ✓ Requires
34830.6 ms ✓ CUDA_Runtime_Discovery
34814.8 ms ✓ Scratch
34814.5 ms ✓ Compat
34697.5 ms ✓ Missings
57520.6 ms ✓ MacroTools
35260.4 ms ✓ Preferences
35853.2 ms ✓ SentinelArrays
23849.2 ms ✓ ScopedValues
58393.0 ms ✓ RandomNumbers
24006.3 ms ✓ Tables
27724.5 ms ✓ Statistics → SparseArraysExt
25339.1 ms ✓ FixedPointNumbers
24200.5 ms ✓ Compat → CompatLinearAlgebraExt
24237.5 ms ✓ Adapt
24009.8 ms ✓ PrecompileTools
24141.2 ms ✓ JLLWrappers
2317.0 ms ✓ Random123
1382.5 ms ✓ Adapt → AdaptSparseArraysExt
1502.4 ms ✓ GPUArraysCore
2052.0 ms ✓ JuliaNVTXCallbacks_jll
2169.3 ms ✓ LibTracyClient_jll
2213.8 ms ✓ NVTX_jll
1920.7 ms ✓ demumble_jll
2908.7 ms ✓ ColorTypes
3158.3 ms ✓ CUDA_Driver_jll
3327.4 ms ✓ StringManipulation
4461.0 ms ✓ DataStructures
2438.7 ms ✓ Tracy
1688.6 ms ✓ SortingAlgorithms
5896.3 ms ✓ LLVMExtra_jll
8842.3 ms ✓ Colors
11471.5 ms ✓ StaticArrays
1772.4 ms ✓ StaticArrays → StaticArraysStatisticsExt
10097.0 ms ✓ CUDA_Compiler_jll
1894.2 ms ✓ Adapt → AdaptStaticArraysExt
10295.1 ms ✓ CUDA_Runtime_jll
4037.1 ms ✓ NVTX
5371.3 ms ✓ KernelAbstractions
13073.7 ms ✓ LLVM
1531.6 ms ✓ KernelAbstractions → LinearAlgebraExt
1883.0 ms ✓ KernelAbstractions → SparseArraysExt
2110.4 ms ✓ LLVM → BFloat16sExt
2216.7 ms ✓ GPUToolbox
2664.7 ms ✓ UnsafeAtomics → UnsafeAtomicsLLVM
24253.9 ms ✓ PrettyTables
10364.6 ms ✓ GPUArrays
43883.6 ms ✓ GPUCompiler
82451.0 ms ✓ DataFrames
Info Given CUDA was explicitly requested, output will be shown live e[0K
e[0K Downloading artifact: CUDA_Compiler
e[0K Failure artifact: CUDA_Compiler
e[0K Downloading artifact: CUDA_Compiler
e[0K Failure artifact: CUDA_Compiler
e[0KERROR: LoadError: InitError: Unable to automatically download/install artifact 'CUDA_Compiler' from sources listed in '/home/guptam/.julia/packages/CUDA_Compiler_jll/rkyxr/Artifacts.toml'.
e[0KSources attempted:
e[0K- https://pkg.julialang.org/artifact/cae57872d6a54ed051458e5bd00f3eda51cc2dd5
e[0K Error: RequestError: Failed to connect to pkg.julialang.org port 443 after 0 ms: Couldn't connect to server while requesting https://pkg.julialang.org/artifact/cae57872d6a54ed051458e5bd00f3eda51cc2dd5
e[0K- https://github.com/JuliaBinaryWrappers/CUDA_Compiler_jll.jl/releases/download/CUDA_Compiler-v0.2.0+0/CUDA_Compiler.v0.2.0.x86_64-linux-gnu-cuda+13.tar.gz
e[0K Error: RequestError: Failed to connect to github.com port 443 after 0 ms: Couldn't connect to server while requesting https://github.com/JuliaBinaryWrappers/CUDA_Compiler_jll.jl/releases/download/CUDA_Compiler-v0.2.0+0/CUDA_Compiler.v0.2.0.x86_64-linux-gnu-cuda+13.tar.gz
e[0K
e[0KStacktrace:
e[0K [1] error(s::String)
e[0K @ Base ./error.jl:35
e[0K [2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::IOContext{IO})
e[0K @ Pkg.Artifacts ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Artifacts.jl:499
e[0K [3] _artifact_str(__module__::Module, artifacts_toml::String, name::SubString{String}, path_tail::String, artifact_dict::Dict{String, Any}, hash::Base.SHA1, platform::Base.BinaryPlatforms.Platform, lazyartifacts::Any)
e[0K @ Artifacts ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Artifacts/src/Artifacts.jl:570
e[0K [4] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
e[0K @ Base ./essentials.jl:1055
e[0K [5] invokelatest(::Any, ::Any, ::Vararg{Any})
e[0K @ Base ./essentials.jl:1052
e[0K [6] macro expansion
e[0K @ ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Artifacts/src/Artifacts.jl:726 [inlined]
e[0K [7] find_artifact_dir()
e[0K @ CUDA_Compiler_jll ~/.julia/packages/JLLWrappers/m2Pjh/src/wrapper_generators.jl:13
e[0K [8] __init__()
e[0K @ CUDA_Compiler_jll ~/.julia/packages/CUDA_Compiler_jll/rkyxr/src/wrappers/x86_64-linux-gnu-cuda+13.jl:12
e[0K [9] run_module_init(mod::Module, i::Int64)
e[0K @ Base ./loading.jl:1378
e[0K [10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
e[0K @ Base ./loading.jl:1366
e[0K [11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
e[0K @ Base ./loading.jl:1254
e[0K [12] _include_from_serialized (repeats 2 times)
e[0K @ ./loading.jl:1210 [inlined]
e[0K [13] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
e[0K @ Base ./loading.jl:2057
e[0K [14] _require(pkg::Base.PkgId, env::String)
e[0K @ Base ./loading.jl:2527
e[0K [15] __require_prelocked(uuidkey::Base.PkgId, env::String)
e[0K @ Base ./loading.jl:2388
e[0K [16] #invoke_in_world#3
e[0K @ ./essentials.jl:1089 [inlined]
e[0K [17] invoke_in_world
e[0K @ ./essentials.jl:1086 [inlined]
e[0K [18] _require_prelocked(uuidkey::Base.PkgId, env::String)
e[0K @ Base ./loading.jl:2375
e[0K [19] macro expansion
e[0K @ ./loading.jl:2314 [inlined]
e[0K [20] macro expansion
e[0K @ ./lock.jl:273 [inlined]
e[0K [21] __require(into::Module, mod::Symbol)
e[0K @ Base ./loading.jl:2271
e[0K [22] #invoke_in_world#3
e[0K @ ./essentials.jl:1089 [inlined]
e[0K [23] invoke_in_world
e[0K @ ./essentials.jl:1086 [inlined]
e[0K [24] require(into::Module, mod::Symbol)
e[0K @ Base ./loading.jl:2260
e[0K [25] include
e[0K @ ./Base.jl:562 [inlined]
e[0K [26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
e[0K @ Base ./loading.jl:2881
e[0K [27] top-level scope
e[0K @ stdin:6
e[0Kduring initialization of module CUDA_Compiler_jll
e[0Kin expression starting at /home/guptam/.julia/packages/CUDA/Wfi8S/src/CUDA.jl:1
e[0Kin expression starting at stdin:6
✗ CUDA
✗ Atomix → AtomixCUDAExt
70 dependencies successfully precompiled in 240 seconds. 29 already precompiled.
ERROR: LoadError: The following 1 direct dependency failed to precompile:
CUDA
Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/home/guptam/.julia/compiled/v1.11/CUDA/jl_ksp1zR".
Downloading artifact: CUDA_Compiler
Failure artifact: CUDA_Compiler
Downloading artifact: CUDA_Compiler
Failure artifact: CUDA_Compiler
ERROR: LoadError: InitError: Unable to automatically download/install artifact 'CUDA_Compiler' from sources listed in '/home/guptam/.julia/packages/CUDA_Compiler_jll/rkyxr/Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/cae57872d6a54ed051458e5bd00f3eda51cc2dd5
Error: RequestError: Failed to connect to pkg.julialang.org port 443 after 0 ms: Couldn't connect to server while requesting https://pkg.julialang.org/artifact/cae57872d6a54ed051458e5bd00f3eda51cc2dd5
- https://github.com/JuliaBinaryWrappers/CUDA_Compiler_jll.jl/releases/download/CUDA_Compiler-v0.2.0+0/CUDA_Compiler.v0.2.0.x86_64-linux-gnu-cuda+13.tar.gz
Error: RequestError: Failed to connect to github.com port 443 after 0 ms: Couldn't connect to server while requesting https://github.com/JuliaBinaryWrappers/CUDA_Compiler_jll.jl/releases/download/CUDA_Compiler-v0.2.0+0/CUDA_Compiler.v0.2.0.x86_64-linux-gnu-cuda+13.tar.gz
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] ensure_artifact_installed(name::String, meta::Dict{String, Any}, artifacts_toml::String; platform::Base.BinaryPlatforms.Platform, verbose::Bool, quiet_download::Bool, io::IOContext{IO})
@ Pkg.Artifacts ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Pkg/src/Artifacts.jl:499
[3] _artifact_str(__module__::Module, artifacts_toml::String, name::SubString{String}, path_tail::String, artifact_dict::Dict{String, Any}, hash::Base.SHA1, platform::Base.BinaryPlatforms.Platform, lazyartifacts::Any)
@ Artifacts ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Artifacts/src/Artifacts.jl:570
[4] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
@ Base ./essentials.jl:1055
[5] invokelatest(::Any, ::Any, ::Vararg{Any})
@ Base ./essentials.jl:1052
[6] macro expansion
@ ~/.julia/juliaup/julia-1.11.6+0.x64.linux.gnu/share/julia/stdlib/v1.11/Artifacts/src/Artifacts.jl:726 [inlined]
[7] find_artifact_dir()
@ CUDA_Compiler_jll ~/.julia/packages/JLLWrappers/m2Pjh/src/wrapper_generators.jl:13
[8] __init__()
@ CUDA_Compiler_jll ~/.julia/packages/CUDA_Compiler_jll/rkyxr/src/wrappers/x86_64-linux-gnu-cuda+13.jl:12
[9] run_module_init(mod::Module, i::Int64)
@ Base ./loading.jl:1378
[10] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
@ Base ./loading.jl:1366
[11] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
@ Base ./loading.jl:1254
[12] _include_from_serialized (repeats 2 times)
@ ./loading.jl:1210 [inlined]
[13] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
@ Base ./loading.jl:2057
[14] _require(pkg::Base.PkgId, env::String)
@ Base ./loading.jl:2527
[15] __require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2388
[16] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[17] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[18] _require_prelocked(uuidkey::Base.PkgId, env::String)
@ Base ./loading.jl:2375
[19] macro expansion
@ ./loading.jl:2314 [inlined]
[20] macro expansion
@ ./lock.jl:273 [inlined]
[21] __require(into::Module, mod::Symbol)
@ Base ./loading.jl:2271
[22] #invoke_in_world#3
@ ./essentials.jl:1089 [inlined]
[23] invoke_in_world
@ ./essentials.jl:1086 [inlined]
[24] require(into::Module, mod::Symbol)
@ Base ./loading.jl:2260
[25] include
@ ./Base.jl:562 [inlined]
[26] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
@ Base ./loading.jl:2881
[27] top-level scope
@ stdin:6
during initialization of module CUDA_Compiler_jll
in expression starting at /home/guptam/.julia/packages/CUDA/Wfi8S/src/CUDA.jl:1
in expression starting at stdin:
in expression starting at /home/guptam/test_CUDA.jl:112
srun: error: gpu001: task 0: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=7933552.0
Here are the steps I took to get to this stage:
From a fresh install of Julia v1.11.6, I added CUDA as a package and performed ‘using CUDA’ on the head node of the cluster (CPU). I then set the runtime version as follows
CUDA.set_runtime_version!(v"12.9")
I noted the relevant CUDA version by executing nvidia-sim in a bash script.
I then redid ‘using CUDA’ on the head node. This was successful and it downloaded the relevant artifact.
I then tried to run using CUDA on the GPU node. This is what fails with the error pasted above. Our GPU nodes are not allowed to access the internet, so cannot download the relevant artifact. I thought that the artifact had already been downloaded so I am not sure why it needs to do so again. Note that I tried specifying different CUDA versions above (e.g. 11.6) but got the same result.
Any help would be appreciated!