Yolo.jl [CUDA] related issue

n90 · November 22, 2021, 5:37am

I am working on a server with Tesla M60. When i try running the sample code am getting the following error. Using CUDA.jl I checked the versioninfo() and gpu is recognized and working. What is the cause of this error and possible solution for the error.

WARNING: could not import CUDNN.cudnnTensorDescriptor into CUDNN
WARNING: could not import CUDNN.cudnnFilterDescriptor into CUDNN
WARNING: could not import CUDNN.cudnnActivationForwardAD into CUDNN
WARNING: could not import CUDNN.cudnnSoftmaxForwardAD into CUDNN
WARNING: could not import CUDNN.cudnnPoolingForwardAD into CUDNN
WARNING: could not import CUDNN.cudnnDropoutForwardAD into CUDNN
WARNING: could not import CUDNN.cudnnRNNForwardAD into CUDNN
ERROR: LoadError: LoadError: LoadError: UndefVarError: cudnnRNNTempSpaceSizes not defined

jw3126 · November 22, 2021, 7:13am

I don’t know the Yolo.jl package. But the first thing I would guess is that it might be incompatible with the CUDA.jl version installed. Try ]st to find out the CUDA.jl version. Also, you can try ]test CUDA to see if CUDNN workers as expected.

n90 · November 23, 2021, 12:02pm

Hey thanks for the reply. I tried ]test CUDA and it is failing. CUDA.versioninfo() shows CUDNN in its dependencies. I am currently trying to find out what the issue is with CUDA package. Other than ]test CUDA is there a way to check if the CUDA package is initialized properly. Thanks.

maleadt · November 23, 2021, 12:04pm

If CUDA.versioninfo() works, the package initializes successfully. Try creating a CuArray to verify.

n90 · November 26, 2021, 5:47am

Hey thanks for the reply. I created a CuArray and checked. It is working but ]test is still failing. Is there any other way I can traceback the issue? I am using CUDA v3.5.0.

maleadt · November 26, 2021, 7:27am

What is the exact failure? There’s thousands of tests, and it’s possible that some fail on an untested platform. You can file an issue for this.

n90 · November 29, 2021, 12:30pm

On running ]test CUDA it fails with this error.

Testing Running tests…
ERROR: LoadError: LoadError: UndefVarError: parameters not defined
Stacktrace:
[1] call!(builder::LLVM.Builder, rt::GPUCompiler.Runtime.RuntimeMethodInstance, args::Vector{LLVM.ConstantExpr})
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\rtlib.jl:57
[2] emit_exception!(builder::LLVM.Builder, name::String, inst::LLVM.CallInst)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\irgen.jl:262
[3] macro expansion
@ C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\irgen.jl:212 [inlined]
[4] macro expansion
@ C:\Users\GPU.julia\packages\TimerOutputs\SSeq1\src\TimerOutput.jl:252 [inlined]
[5] lower_throw!(mod::LLVM.Module)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\irgen.jl:173
[6] module_pass_callback(ptr::Ptr{Nothing}, data::Ptr{Nothing})
@ LLVM C:\Users\GPU.julia\packages\LLVM\srSVa\src\pass.jl:21
[7] macro expansion
@ C:\Users\GPU.julia\packages\LLVM\srSVa\src\util.jl:85 [inlined]
[8] LLVMRunPassManager
@ C:\Users\GPU.julia\packages\LLVM\srSVa\lib\11\libLLVM_h.jl:4707 [inlined]
[9] run!
@ C:\Users\GPU.julia\packages\LLVM\srSVa\src\passmanager.jl:39 [inlined]
[10] (::GPUCompiler.var"#51#53"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(GPUCompiler.Runtime.box_bool), Tuple{Bool}}}, LLVM.Module})(pm::LLVM.ModulePassManager)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\irgen.jl:82
[11] LLVM.ModulePassManager(::GPUCompiler.var"#51#53"{GPUCompiler.CompilerJob{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{typeof(GPUCompiler.Runtime.box_bool), Tuple{Bool}}}, LLVM.Module})
@ LLVM C:\Users\GPU.julia\packages\LLVM\srSVa\src\passmanager.jl:33
[12] macro expansion
@ C:\Users\GPU.julia\packages\TimerOutputs\SSeq1\src\TimerOutput.jl:252 [inlined]
[13] irgen(job::GPUCompiler.CompilerJob, method_instance::Core.MethodInstance, world::UInt64)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\irgen.jl:65
[14] macro expansion
@ C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\driver.jl:130 [inlined]
[15] macro expansion
@ C:\Users\GPU.julia\packages\TimerOutputs\SSeq1\src\TimerOutput.jl:252 [inlined]
[16] macro expansion
@ C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\driver.jl:129 [inlined]
[17] emit_llvm(job::GPUCompiler.CompilerJob, method_instance::Any, world::UInt64; libraries::Bool, deferred_codegen::Bool, optimize::Bool, only_entry::Bool)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\utils.jl:62
[18] codegen(output::Symbol, job::GPUCompiler.CompilerJob; libraries::Bool, deferred_codegen::Bool, optimize::Bool, strip::Bool, validate::Bool, only_entry::Bool)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\driver.jl:55
[19] emit_function!(mod::LLVM.Module, job::GPUCompiler.CompilerJob, f::Function, method::GPUCompiler.Runtime.RuntimeMethodInstance)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\rtlib.jl:76
[20] build_runtime(job::GPUCompiler.CompilerJob, ctx::LLVM.Context)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\rtlib.jl:126
[21] load_runtime(job::GPUCompiler.CompilerJob, ctx::LLVM.Context)
@ GPUCompiler C:\Users\GPU.julia\packages\GPUCompiler\XwWPj\src\rtlib.jl:174
[22] (::CUDA.var"#186#188"{Set{VersionNumber}, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{CUDA.var"#185#187", Tuple{}}})(ctx::LLVM.Context)
@ CUDA C:\Users\GPU.julia\packages\CUDA\M4jkK\src\device\runtime.jl:21
[23] LLVM.Context(f::CUDA.var"#186#188"{Set{VersionNumber}, CUDA.CUDACompilerParams, GPUCompiler.FunctionSpec{CUDA.var"#185#187", Tuple{}}})
@ LLVM C:\Users\GPU.julia\packages\LLVM\srSVa\src\core\context.jl:22
[24] JuliaContext
@ C:\Users\GPU.julia\packages\LLVM\srSVa\src\interop\base.jl:29 [inlined]
[25] precompile_runtime
@ C:\Users\GPU.julia\packages\CUDA\M4jkK\src\device\runtime.jl:15 [inlined]
[26] precompile_runtime()
@ CUDA C:\Users\GPU.julia\packages\CUDA\M4jkK\src\device\runtime.jl:13
[27] top-level scope
@ C:\Users\GPU.julia\packages\CUDA\M4jkK\test\setup.jl:23
[28] include(fname::String)
@ Base.MainInclude .\client.jl:444
[29] top-level scope
@ C:\Users\GPU.julia\packages\CUDA\M4jkK\test\runtests.jl:51
[30] include(fname::String)
@ Base.MainInclude .\client.jl:444
[31] top-level scope
@ none:6
in expression starting at C:\Users\GPU.julia\packages\CUDA\M4jkK\test\setup.jl:23
in expression starting at C:\Users\GPU.julia\packages\CUDA\M4jkK\test\runtests.jl:51
ERROR: Package CUDA errored during testing

maleadt · November 29, 2021, 1:13pm

That looks like Both ExprTools and LLVM export "parameters" · Issue #214 · JuliaGPU/GPUCompiler.jl · GitHub – you’re probably using outdated packages.

cirobr · April 24, 2022, 11:17am

Had exactly same issue. Found that CUDA was outdated and LLVM not installed. Solution was simply:

]
add LLVM
up CUDA

After that, CUDA testing ran properly.