I cannot use my gpu despite trying everything

kocak · January 5, 2024, 5:54pm

I get this error sadly, I checked online and tried with a litle different syntax but it gave the second error instead:

EDIT:
As I am a new user, I am notified I have to wait 6 hours before I can write replies again which will be insanely late where I am. So I will be forced to reply you tomorow. I just wanted to notify this.

ufechner7 · January 5, 2024, 5:56pm

Well, I can test this myself only when I am back home at my desktop computer, so you have to wait a little bit…

JoshuaBillson · January 6, 2024, 5:28am

I want to add that I’m currently having an identical problem on Julia 1.9.1 and Flux 0.14.8.

Here’s a minimum working example:

using CUDA, cuDNN, Flux

X = rand(Float32, 128, 128, 3, 16)
Y = rand(Float32, 128, 128, 128, 16)
	
data = Flux.Data.DataLoader((X, Y), batchsize=1) |> gpu

model = Conv((3,3), 3=>128, relu, pad=SamePad()) |> gpu

opt = Flux.setup(Adam(), model)
	
Flux.train!(model, data, opt) do m, x, y
	Flux.mae(m(x), y)
end

Running this throws an error as soon as the training loop tries to execute. CUDA is complaining about an illegal memory access and the following is displayed in the terminal:

WARNING: Error while freeing DeviceBuffer(8.000 MiB at 0x000000150503ac00):
CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), details=CUDA.Optional{String}(data=nothing))

Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/libcuda.jl:27
  [2] check
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/libcuda.jl:34 [inlined]
  [3] cuMemFreeAsync
    @ ~/.julia/packages/CUDA/YIj5X/lib/utils/call.jl:26 [inlined]
  [4] #free#2
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:97 [inlined]
  [5] free
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:92 [inlined]
  [6] #actual_free#1003
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:78 [inlined]
  [7] actual_free
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:75 [inlined]
  [8] #_free#1028
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:506 [inlined]
  [9] _free
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:493 [inlined]
 [10] macro expansion
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:478 [inlined]
 [11] macro expansion
    @ ./timing.jl:393 [inlined]
 [12] #free#1027
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:477 [inlined]
 [13] free
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:466 [inlined]
 [14] (::CUDA.var"#1034#1035"{CUDA.Mem.DeviceBuffer, Bool})()
    @ CUDA ~/.julia/packages/CUDA/YIj5X/src/array.jl:101
 [15] #context!#915
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/state.jl:170 [inlined]
 [16] context!
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/state.jl:165 [inlined]
 [17] _free_buffer(buf::CUDA.Mem.DeviceBuffer, early::Bool)
    @ CUDA ~/.julia/packages/CUDA/YIj5X/src/array.jl:89
 [18] release(rc::GPUArrays.RefCounted{CUDA.Mem.DeviceBuffer}, args::Bool)
    @ GPUArrays ~/.julia/packages/GPUArrays/dAUOE/src/host/abstractarray.jl:42
 [19] unsafe_free!
    @ ~/.julia/packages/GPUArrays/dAUOE/src/host/abstractarray.jl:90 [inlined]
 [20] unsafe_finalize!(xs::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer})
    @ CUDA ~/.julia/packages/CUDA/YIj5X/src/array.jl:113
 [21] #invokelatest#2
    @ ./essentials.jl:818 [inlined]
 [22] invokelatest
    @ ./essentials.jl:813 [inlined]
 [23] macro expansion
    @ ./logging.jl:365 [inlined]
 [24] macro expansion
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:385 [inlined]
 [25] train!(loss::Function, model::Flux.Conv{2, 4, typeof(NNlib.relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, data::MLUtils.DataLoader{MLUtils.MappedData{:auto, typeof(Flux.gpu), Tuple{Array{Float32, 4}, Array{Float32, 4}}}, Random._GLOBAL_RNG, Val{nothing}}, opt::NamedTuple{(:σ, :weight, :bias, :stride, :pad, :dilation, :groups), Tuple{Tuple{}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Tuple{Tuple{}, Tuple{}}, NTuple{4, Tuple{}}, Tuple{Tuple{}, Tuple{}}, Tuple{}}}; cb::Nothing)
    @ Flux.Train ~/.julia/packages/Flux/PpGmk/src/train.jl:105
 [26] train!(loss::Function, model::Flux.Conv{2, 4, typeof(NNlib.relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, data::MLUtils.DataLoader{MLUtils.MappedData{:auto, typeof(Flux.gpu), Tuple{Array{Float32, 4}, Array{Float32, 4}}}, Random._GLOBAL_RNG, Val{nothing}}, opt::NamedTuple{(:σ, :weight, :bias, :stride, :pad, :dilation, :groups), Tuple{Tuple{}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Tuple{Tuple{}, Tuple{}}, NTuple{4, Tuple{}}, Tuple{Tuple{}, Tuple{}}, Tuple{}}})
    @ Flux.Train ~/.julia/packages/Flux/PpGmk/src/train.jl:102
 [27] ##function_wrapped_cell#297
    @ ~/Julia/CatsVsDogs/Pansharpening/notebool.jl#==#e832870d-ec3f-4f22-92fa-1e26a954a67e:11 [inlined]
 [28] var"##function_wrapped_cell#297"(Conv::Type{Flux.Conv}, Adam::Type{Flux.Optimise.Adam}, SamePad::Type{Flux.SamePad}, |>::typeof(|>), relu::Function, Float32::Type, rand::typeof(rand), Flux::Module, gpu::typeof(Flux.gpu), =>::Type)
    @ Main ./none:0
 [29] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:816
 [30] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:813
 [31] compute(m::Module, computer::Main.PlutoRunner.Computer)
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:434
 [32] (::Main.PlutoRunner.var"#37#41"{Module})()
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:610
 [33] run_inside_trycatch(m::Module, f::Main.PlutoRunner.var"#37#41"{Module})
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:474
 [34] (::Main.PlutoRunner.var"#35#39"{Bool, Module, Expr, Base.UUID, Base.UUID, Tuple{Set{Symbol}, Set{Symbol}}, Nothing, UInt64, Expr})()
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:610
 [35] with_io_to_logs(f::Main.PlutoRunner.var"#35#39"{Bool, Module, Expr, Base.UUID, Base.UUID, Tuple{Set{Symbol}, Set{Symbol}}, Nothing, UInt64, Expr}; enabled::Bool, loglevel::Base.CoreLogging.LogLevel)
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2476
 [36] with_io_to_logs
    @ ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2426 [inlined]
 [37] (::Main.PlutoRunner.var"#123#124"{Bool, Base.CoreLogging.LogLevel, Main.PlutoRunner.var"#35#39"{Bool, Module, Expr, Base.UUID, Base.UUID, Tuple{Set{Symbol}, Set{Symbol}}, Nothing, UInt64, Expr}})()
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2500
 [38] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:514
 [39] with_logger
    @ ./logging.jl:626 [inlined]
 [40] #with_logger_and_io_to_logs#122
    @ ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2499 [inlined]
 [41] with_logger_and_io_to_logs
    @ ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2498 [inlined]
 [42] run_expression(m::Module, expr::Expr, notebook_id::Base.UUID, cell_id::Base.UUID, function_wrapped_info::Union{Nothing, Tuple{Set{Symbol}, Set{Symbol}}}, forced_expr_id::Union{Nothing, UInt64}; user_requested_run::Bool, capture_stdout::Bool)
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:585
 [43] top-level scope
    @ ~/.julia/packages/Pluto/YRKRh/src/evaluation/WorkspaceManager.jl:457
 [44] eval(m::Module, e::Any)
    @ Core ./boot.jl:370
 [45] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:816
 [46] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:813
 [47] (::Distributed.var"#114#116"{Distributed.CallWaitMsg})()
    @ Distributed ~/Julia/bin/julia-1.9.1/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:301
 [48] run_work_thunk(thunk::Distributed.var"#114#116"{Distributed.CallWaitMsg}, print_error::Bool)
    @ Distributed ~/Julia/bin/julia-1.9.1/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
 [49] run_work_thunk(rv::Distributed.RemoteValue, thunk::Function)
    @ Distributed ~/Julia/bin/julia-1.9.1/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
 [50] (::Distributed.var"#100#102"{Distributed.RemoteValue, Distributed.var"#114#116"{Distributed.CallWaitMsg}})()
    @ Distributed ./task.jl:514
error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), details=CUDA.Optional{String}(data=nothing))
throw_api_error at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/libcuda.jl:27
check at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/libcuda.jl:34 [inlined]
cuMemHostUnregister at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/utils/call.jl:26 [inlined]
unregister at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:193 [inlined]
#21 at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:701 [inlined]
#context!#915 at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/state.jl:170 [inlined]
context! at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/state.jl:165 [inlined]
macro expansion at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:700 [inlined]
macro expansion at ./lock.jl:267 [inlined]
__unpin at /home/jmbillson/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:694
#1083 at /home/jmbillson/.julia/packages/CUDA/YIj5X/src/compiler/execution.jl:180
unknown function (ip: 0x7f16c991bcb2)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
run_finalizer at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gc.c:417
jl_gc_run_finalizers_in_list at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gc.c:507
run_finalizers at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gc.c:553
jl_mutex_unlock at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia_locks.h:81 [inlined]
jl_generate_fptr_impl at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/jitlayers.cpp:467
jl_compile_method_internal at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2348 [inlined]
jl_compile_method_internal at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2237
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2750 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:818 [inlined]
invokelatest at ./essentials.jl:813 [inlined]
macro expansion at ./logging.jl:365 [inlined]
macro expansion at /home/jmbillson/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:385 [inlined]
#train!#3 at /home/jmbillson/.julia/packages/Flux/PpGmk/src/train.jl:105
train! at /home/jmbillson/.julia/packages/Flux/PpGmk/src/train.jl:102
unknown function (ip: 0x7f17ded233c2)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
##function_wrapped_cell#297 at /home/jmbillson/Julia/CatsVsDogs/Pansharpening/notebool.jl#==#e832870d-ec3f-4f22-92fa-1e26a954a67e:11 [inlined]
##function_wrapped_cell#297 at ./none:0
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:774
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:730
#invokelatest#2 at ./essentials.jl:816
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:730
invokelatest at ./essentials.jl:813
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:730
compute at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:434
#37 at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:610
run_inside_trycatch at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:474
#35 at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:610
#with_io_to_logs#119 at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2476
with_io_to_logs at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2426 [inlined]
#123 at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2500
unknown function (ip: 0x7f17ded1178f)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
with_logstate at ./logging.jl:514
with_logger at ./logging.jl:626 [inlined]
#with_logger_and_io_to_logs#122 at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2499 [inlined]
with_logger_and_io_to_logs at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2498 [inlined]
#run_expression#33 at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:585
unknown function (ip: 0x7f17ded0a5bd)
run_expression at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:503
unknown function (ip: 0x7f17ded07496)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_stmt_value at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
eval_body at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/interpreter.c:762
top-level scope at /home/jmbillson/.julia/packages/Pluto/YRKRh/src/evaluation/WorkspaceManager.jl:457
jl_toplevel_eval_flex at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:912
ijl_toplevel_eval_in at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370
jfptr_eval_12237.clone_1 at /home/jmbillson/Julia/bin/julia-1.9.1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:774
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:730
#invokelatest#2 at ./essentials.jl:816
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:730
invokelatest at ./essentials.jl:813
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/builtins.c:730
#114 at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:301
run_work_thunk at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
unknown function (ip: 0x7f17dec80509)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
run_work_thunk at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
#100 at ./task.jl:514
unknown function (ip: 0x7f17dec800cf)
_jl_invoke at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
start_task at /cache/build/default-amdci4-6/julialang/julia-release-1-dot-9/src/task.c:1092
WARNING: Error while freeing DeviceBuffer(8.000 MiB at 0x000000150483ac00):
CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), details=CUDA.Optional{String}(data=nothing))

Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/libcuda.jl:27
  [2] check
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/libcuda.jl:34 [inlined]
  [3] cuMemFreeAsync
    @ ~/.julia/packages/CUDA/YIj5X/lib/utils/call.jl:26 [inlined]
  [4] #free#2
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:97 [inlined]
  [5] free
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/memory.jl:92 [inlined]
  [6] #actual_free#1003
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:78 [inlined]
  [7] actual_free
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:75 [inlined]
  [8] #_free#1028
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:506 [inlined]
  [9] _free
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:493 [inlined]
 [10] macro expansion
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:478 [inlined]
 [11] macro expansion
    @ ./timing.jl:393 [inlined]
 [12] #free#1027
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:477 [inlined]
 [13] free
    @ ~/.julia/packages/CUDA/YIj5X/src/pool.jl:466 [inlined]
 [14] (::CUDA.var"#1034#1035"{CUDA.Mem.DeviceBuffer, Bool})()
    @ CUDA ~/.julia/packages/CUDA/YIj5X/src/array.jl:101
 [15] #context!#915
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/state.jl:170 [inlined]
 [16] context!
    @ ~/.julia/packages/CUDA/YIj5X/lib/cudadrv/state.jl:165 [inlined]
 [17] _free_buffer(buf::CUDA.Mem.DeviceBuffer, early::Bool)
    @ CUDA ~/.julia/packages/CUDA/YIj5X/src/array.jl:89
 [18] release(rc::GPUArrays.RefCounted{CUDA.Mem.DeviceBuffer}, args::Bool)
    @ GPUArrays ~/.julia/packages/GPUArrays/dAUOE/src/host/abstractarray.jl:42
 [19] unsafe_free!
    @ ~/.julia/packages/GPUArrays/dAUOE/src/host/abstractarray.jl:90 [inlined]
 [20] unsafe_finalize!(xs::CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer})
    @ CUDA ~/.julia/packages/CUDA/YIj5X/src/array.jl:113
 [21] #invokelatest#2
    @ ./essentials.jl:818 [inlined]
 [22] invokelatest
    @ ./essentials.jl:813 [inlined]
 [23] macro expansion
    @ ./logging.jl:365 [inlined]
 [24] macro expansion
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:385 [inlined]
 [25] train!(loss::Function, model::Flux.Conv{2, 4, typeof(NNlib.relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, data::MLUtils.DataLoader{MLUtils.MappedData{:auto, typeof(Flux.gpu), Tuple{Array{Float32, 4}, Array{Float32, 4}}}, Random._GLOBAL_RNG, Val{nothing}}, opt::NamedTuple{(:σ, :weight, :bias, :stride, :pad, :dilation, :groups), Tuple{Tuple{}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Tuple{Tuple{}, Tuple{}}, NTuple{4, Tuple{}}, Tuple{Tuple{}, Tuple{}}, Tuple{}}}; cb::Nothing)
    @ Flux.Train ~/.julia/packages/Flux/PpGmk/src/train.jl:105
 [26] train!(loss::Function, model::Flux.Conv{2, 4, typeof(NNlib.relu), CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}, data::MLUtils.DataLoader{MLUtils.MappedData{:auto, typeof(Flux.gpu), Tuple{Array{Float32, 4}, Array{Float32, 4}}}, Random._GLOBAL_RNG, Val{nothing}}, opt::NamedTuple{(:σ, :weight, :bias, :stride, :pad, :dilation, :groups), Tuple{Tuple{}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 4, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Optimisers.Leaf{Optimisers.Adam, Tuple{CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CUDA.CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, Tuple{Float32, Float32}}}, Tuple{Tuple{}, Tuple{}}, NTuple{4, Tuple{}}, Tuple{Tuple{}, Tuple{}}, Tuple{}}})
    @ Flux.Train ~/.julia/packages/Flux/PpGmk/src/train.jl:102
 [27] ##function_wrapped_cell#297
    @ ~/Julia/CatsVsDogs/Pansharpening/notebool.jl#==#e832870d-ec3f-4f22-92fa-1e26a954a67e:11 [inlined]
 [28] var"##function_wrapped_cell#297"(Conv::Type{Flux.Conv}, Adam::Type{Flux.Optimise.Adam}, SamePad::Type{Flux.SamePad}, |>::typeof(|>), relu::Function, Float32::Type, rand::typeof(rand), Flux::Module, gpu::typeof(Flux.gpu), =>::Type)
    @ Main ./none:0
 [29] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:816
 [30] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:813
 [31] compute(m::Module, computer::Main.PlutoRunner.Computer)
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:434
 [32] (::Main.PlutoRunner.var"#37#41"{Module})()
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:610
 [33] run_inside_trycatch(m::Module, f::Main.PlutoRunner.var"#37#41"{Module})
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:474
 [34] (::Main.PlutoRunner.var"#35#39"{Bool, Module, Expr, Base.UUID, Base.UUID, Tuple{Set{Symbol}, Set{Symbol}}, Nothing, UInt64, Expr})()
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:610
 [35] with_io_to_logs(f::Main.PlutoRunner.var"#35#39"{Bool, Module, Expr, Base.UUID, Base.UUID, Tuple{Set{Symbol}, Set{Symbol}}, Nothing, UInt64, Expr}; enabled::Bool, loglevel::Base.CoreLogging.LogLevel)
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2476
 [36] with_io_to_logs
    @ ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2426 [inlined]
 [37] (::Main.PlutoRunner.var"#123#124"{Bool, Base.CoreLogging.LogLevel, Main.PlutoRunner.var"#35#39"{Bool, Module, Expr, Base.UUID, Base.UUID, Tuple{Set{Symbol}, Set{Symbol}}, Nothing, UInt64, Expr}})()
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2500
 [38] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:514
 [39] with_logger
    @ ./logging.jl:626 [inlined]
 [40] #with_logger_and_io_to_logs#122
    @ ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2499 [inlined]
 [41] with_logger_and_io_to_logs
    @ ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:2498 [inlined]
 [42] run_expression(m::Module, expr::Expr, notebook_id::Base.UUID, cell_id::Base.UUID, function_wrapped_info::Union{Nothing, Tuple{Set{Symbol}, Set{Symbol}}}, forced_expr_id::Union{Nothing, UInt64}; user_requested_run::Bool, capture_stdout::Bool)
    @ Main.PlutoRunner ~/.julia/packages/Pluto/YRKRh/src/runner/PlutoRunner.jl:585
 [43] top-level scope
    @ ~/.julia/packages/Pluto/YRKRh/src/evaluation/WorkspaceManager.jl:457
 [44] eval(m::Module, e::Any)
    @ Core ./boot.jl:370
 [45] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Base ./essentials.jl:816
 [46] invokelatest(::Any, ::Any, ::Vararg{Any})
    @ Base ./essentials.jl:813
 [47] (::Distributed.var"#114#116"{Distributed.CallWaitMsg})()
    @ Distributed ~/Julia/bin/julia-1.9.1/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:301
 [48] run_work_thunk(thunk::Distributed.var"#114#116"{Distributed.CallWaitMsg}, print_error::Bool)
    @ Distributed ~/Julia/bin/julia-1.9.1/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
 [49] run_work_thunk(rv::Distributed.RemoteValue, thunk::Function)
    @ Distributed ~/Julia/bin/julia-1.9.1/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
 [50] (::Distributed.var"#100#102"{Distributed.RemoteValue, Distributed.var"#114#116"{Distributed.CallWaitMsg}})()
    @ Distributed ./task.jl:514

This must be something recent, because I’ve trained on the GPU before without issue.

ToucheSir · January 6, 2024, 5:37am

This may be related to Pullback on mean() gives illegal memory access code 700 · Issue #1473 · FluxML/Zygote.jl · GitHub. Try rolling back to CUDA.jl v5.0.0 as suggested here and see if that works for you.

ufechner7 · January 6, 2024, 12:27pm

I created a bug report for CUDA: Testing of CUDA fails · Issue #2222 · JuliaGPU/CUDA.jl · GitHub

ufechner7 · January 6, 2024, 2:17pm

Using CUDA#master works a little bit better for me. You can add it with:

]
add CUDA#master

When I test this now, only one error remains, see: Tests fail for CUDA#master · Issue #2223 · JuliaGPU/CUDA.jl · GitHub

Can you try this with your Flux example to see if you still get the same error?

ToucheSir · January 6, 2024, 4:39pm

As mentioned in the last thread I thinked, that won’t work because of CompatHelper: bump compat for Adapt to 4, (keep existing compat) by github-actions[bot] · Pull Request #2362 · FluxML/Flux.jl · GitHub. Until Flux’s 3rd party dependencies bump their compats, the best route is unfortunately to roll back CUDA.jl.

maleadt · January 6, 2024, 6:01pm

Seeing how the ML ecosystem is struggling with the Adapt.jl bump, I’ll create a backports release: Backports for v5.1 by maleadt · Pull Request #2224 · JuliaGPU/CUDA.jl · GitHub

kocak · January 7, 2024, 4:55pm

Thanks to this by using

using Pkg
Pkg.update("CUDA")

and confirming

using CUDA
CUDA.versioninfo()

outputs => 5.1.2
I have fixed the problem.
I would like to thank everyone here who have helped me for my problem.

Topic		Replies	Views
CUDAnative.jl failed to initialize GPU	11	1620	February 10, 2020
A newbie can not use CUDA by follow official DOC! GPU	33	6599	October 16, 2020
Free GPU resources? GPU	1	547	June 3, 2020
Error in Cuda function : ERROR: LoadError: CuError(1, nothing) General Usage question , cudanative , cuda	8	1090	July 26, 2019
Can't load CuArrays GPU	17	3087	June 22, 2020

I cannot use my gpu despite trying everything

Related topics