# CUDA\_ERROR\_ILLEGAL\_ADDRESS with CuArrays/Zygote getindex

**URL:** <https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744>\
**Category:** GPU\
**Created:** [December 27, 2019, 7:00pm UTC](https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744 "2019-12-27T19:00:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [December 27, 2019, 7:00pm UTC](https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744/1 "2019-12-27T19:00:56Z")

</div>

Hello, I am getting an error CuError(CUDA\_ERROR\_ILLEGAL\_ADDRESS) with the following stacktrace. My code is trying to take gradients of a deep learning model with `Zygote`. I wonder if this is a symptom of running out of memory, or a bug in CuArrays/Zygote?

The line where it fails in `Zygote` corresponds to `∇getindex`. My code is indexing into a `CuArray` using an array of integers in the main memory, like this:

```julia
x[idc, :] # typeof(x) == CuArray{Float32, 2}; typeof(idc) == Vector{Int}

```

```julia
ERROR: LoadError: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
Stacktrace:
 [1] cuMemcpyHtoD_v2(::CUDAdrv.CuPtr{Int64}, ::Ptr{Int64}, ::Int64) at /rds/general/user/et517/home/.julia/packages/CUDAdrv/3EzC1/src/error.jl:123
 [2] #unsafe_copyto!#6 at /rds/general/user/et517/home/.julia/packages/CUDAdrv/3EzC1/src/memory.jl:285 [inlined]
 [3] unsafe_copyto! at /rds/general/user/et517/home/.julia/packages/CUDAdrv/3EzC1/src/memory.jl:278 [inlined]
 [4] copyto! at /rds/general/user/et517/home/.julia/packages/CuArrays/ZYCpV/src/array.jl:254 [inlined]
 [5] copyto!(::CuArrays.CuArray{Int64,1,Nothing}, ::Array{Int64,1}) at /rds/general/user/et517/home/.julia/packages/GPUArrays/1wgPO/src/abstractarray.jl:118
 [6] convert(::Type{CuArrays.CuArray}, ::Array{Int64,1}) at /rds/general/user/et517/home/.julia/packages/GPUArrays/1wgPO/src/construction.jl:84
 [7] _adapt_structure at /rds/general/user/et517/home/.julia/packages/CuArrays/ZYCpV/src/array.jl:237 [inlined]
 [8] adapt_structure at /rds/general/user/et517/home/.julia/packages/Adapt/aeQPS/src/base.jl:12 [inlined]
 [9] adapt at /rds/general/user/et517/home/.julia/packages/Adapt/aeQPS/src/Adapt.jl:6 [inlined]
 [10] adapt_structure(::CUDAnative.Adaptor, ::SubArray{Float32,2,CuArrays.CuArray{Float32,2,Nothing},Tuple{Array{Int64,1},Base.Slice{Base.OneTo{Int64}}},false}) at /rds/general/user/et517/home/.julia/packages/CuArrays/ZYCpV/src/subarray.jl:63
 [11] adapt at /rds/general/user/et517/home/.julia/packages/Adapt/aeQPS/src/Adapt.jl:6 [inlined]
 [12] cudaconvert at /rds/general/user/et517/home/.julia/packages/CUDAnative/RhbZ0/src/execution.jl:211 [inlined]
 [13] map at ./tuple.jl:141 [inlined]
 [14] macro expansion at /rds/general/user/et517/home/.julia/packages/CUDAnative/RhbZ0/src/execution.jl:174 [inlined]
 [15] macro expansion at ./gcutils.jl:87 [inlined]
 [16] macro expansion at /rds/general/user/et517/home/.julia/packages/CUDAnative/RhbZ0/src/execution.jl:173 [inlined]
 [17] _gpu_call(::CuArrays.CuArrayBackend, ::Function, ::SubArray{Float32,2,CuArrays.CuArray{Float32,2,Nothing},Tuple{Array{Int64,1},Base.Slice{Base.OneTo{Int64}}},false}, ::Tuple{SubArray{Float32,2,CuArrays.CuArray{Float32,2,Nothing},Tuple{Array{Int64,1},Base.Slice{Base.OneTo{Int64}}},false},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}},typeof(+),Tuple{Base.Broadcast.Extruded{SubArray{Float32,2,CuArrays.CuArray{Float32,2,Nothing},Tuple{Array{Int64,1},Base.Slice{Base.OneTo{Int64}}},false},Tuple{Bool,Bool},Tuple{Int64,Int64}},Base.Broadcast.Extruded{CuArrays.CuArray{Float32,2,Nothing},Tuple{Bool,Bool},Tuple{Int64,Int64}}}}}, ::Tuple{Tuple{Int64},Tuple{Int64}}) at /rds/general/user/et517/home/.julia/packages/CuArrays/ZYCpV/src/gpuarray_interface.jl:62
 [18] gpu_call at /rds/general/user/et517/home/.julia/packages/GPUArrays/1wgPO/src/abstract_gpu_interface.jl:151 [inlined]
 [19] gpu_call at /rds/general/user/et517/home/.julia/packages/GPUArrays/1wgPO/src/abstract_gpu_interface.jl:128 [inlined]
 [20] copyto! at /rds/general/user/et517/home/.julia/packages/GPUArrays/1wgPO/src/broadcast.jl:48 [inlined]
 [21] copyto! at ./broadcast.jl:842 [inlined]
 [22] materialize! at ./broadcast.jl:801 [inlined]
 [23] (::getfield(Zygote, Symbol("##984#986")){CuArrays.CuArray{Float32,2,CuArrays.CuArray{Float32,4,Nothing}},Tuple{Array{Int64,1},Colon}})(::CuArrays.CuArray{Float32,2,Nothing}) at /rds/general/user/et517/home/.julia/packages/Zygote/N2BNN/src/lib/array.jl:38
...

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [December 28, 2019, 1:12pm UTC](https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744/2 "2019-12-28T13:12:53Z")

</div>

Don’t you see a kernel exception some time earlier? If you do, you can run with `julia -g2` to see more details. If you don’t, try running with `--check-bounds=yes`.

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [January 2, 2020, 6:20pm UTC](https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744/3 "2020-01-02T18:20:53Z")

</div>

I get lots of error messages like this:

```julia
error in running finalizer: CUDAdrv.CuError(code=CUDAdrv.cudaError_enum(0x000002bc), meta=nothing)

```

I have tried running `julia -g2 --check-bounds=yes` but got the same error.

This must be somehow related to it running out of memory, because when I reduce the minibatch size it goes away.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 3, 2020, 6:48am UTC](https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744/4 "2020-01-03T06:48:37Z")

</div>

CUDA errors persist, so the ones you are seeing in the finalizer are the same you caught earlier one time (as reported in your first post), i.e., CUDA\_ERROR\_ILLEGAL\_ADDRESS.

Are you switching devices, perhaps?

---

<div class="post-metadata">

**Author:** ![tanhevg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tanhevg/32/12025_2.png) [@tanhevg](https://discourse.julialang.org/u/tanhevg)\
**Post date:** [January 3, 2020, 12:07pm UTC](https://discourse.julialang.org/t/cuda-error-illegal-address-with-cuarrays-zygote-getindex/32744/5 "2020-01-03T12:07:56Z")

</div>

I am not switching devices, although the server where I run this and get the error has multiple GPUs.

When I run this on a less powerful machine with a single GPU and less memory, I get a nice “Out of GPU memory” error.