# Flux.train failing with CUDA GPU

**URL:** <https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003>\
**Category:** Machine Learning\
**Created:** [March 18, 2019, 4:40pm UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003 "2019-03-18T16:40:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bjenkinsgit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjenkinsgit/32/3440_2.png) [@bjenkinsgit](https://discourse.julialang.org/u/bjenkinsgit)\
**Post date:** [March 18, 2019, 4:40pm UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003/1 "2019-03-18T16:40:21Z")

</div>

I’m testing some julia Deep Learning code and at the point of invoking Flux.train, I get the following:

ReadOnlyMemoryError()

specifically, I’m invoking the train function call like so:

@time for i in 1:100  
Flux.train!(loss, params, data, opt)  
end

and I get the following stack trace:

Stacktrace:  
[1] gemv!(::Char, ::Float32, ::CuArray{Float32,2}, ::Array{Float32,1}, ::Float32, ::Array{Float32,1}) at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/blas.jl:577  
[2] gemv!(::Array{Float32,1}, ::Char, ::CuArray{Float32,2}, ::Array{Float32,1}) at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:360  
[3] \* at /buildworker/worker/package\_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:64 [inlined]  
[4] \_forward at /home/bwj/.julia/packages/Flux/8XpDt/src/tracker/lib/array.jl:361 [inlined]  
[5] #track#1 at /home/bwj/.julia/packages/Flux/8XpDt/src/tracker/Tracker.jl:51 [inlined]  
[6] track at /home/bwj/.julia/packages/Flux/8XpDt/src/tracker/Tracker.jl:51 [inlined]  
[7] \* at /home/bwj/.julia/packages/Flux/8XpDt/src/tracker/lib/array.jl:353 [inlined]  
[8] Dense at /home/bwj/.julia/packages/Flux/8XpDt/src/layers/basic.jl:82 [inlined]  
[9] Dense at /home/bwj/.julia/packages/Flux/8XpDt/src/layers/basic.jl:122 [inlined]  
[10] (::Dense{typeof(σ),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}})(::Array{Float64,1}) at /home/bwj/.julia/packages/Flux/8XpDt/src/layers/basic.jl:125  
[11] applychain(::Tuple{Dense{typeof(σ),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}},Dense{typeof(σ),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}}, ::Array{Float64,1}) at /home/bwj/.julia/packages/Flux/8XpDt/src/layers/basic.jl:31  
[12] Chain at /home/bwj/.julia/packages/Flux/8XpDt/src/layers/basic.jl:33 [inlined]  
[13] loss(::Array{Float64,1}, ::Flux.OneHotVector) at ./In[13]:1  
[14] macro expansion at /home/bwj/.julia/packages/Flux/8XpDt/src/optimise/train.jl:74 [inlined]  
[15] macro expansion at /home/bwj/.julia/packages/Juno/B1s6e/src/progress.jl:133 [inlined]  
[16] #train!#12(::getfield(Flux.Optimise, Symbol(“##14#18”)), ::Function, ::Function, ::Function, ::Base.Iterators.Zip{Tuple{Array{Array{Float64,1},1},Array{Flux.OneHotVector,1}}}, ::Function) at /home/bwj/.julia/packages/Flux/8XpDt/src/optimise/train.jl:72  
[17] train!(::Function, ::Function, ::Base.Iterators.Zip{Tuple{Array{Array{Float64,1},1},Array{Flux.OneHotVector,1}}}, ::Function) at /home/bwj/.julia/packages/Flux/8XpDt/src/optimise/train.jl:70  
[18] macro expansion at ./In[17]:2 [inlined]  
[19] macro expansion at ./util.jl:156 [inlined]  
[20] top-level scope at ./In[17]:1 [inlined]  
[21] top-level scope at ./none:0

* * *

- I am running on Ubuntu 18.04 LTS
- nvidia-settings reports NVIDIA Driver version: 418.43
- running the MNIST cuDNN outputs:

cudnnGetVersion() : 7500 , CUDNN\_VERSION from cudnn.h : 7500 (7.5.0)  
Host compiler version : GCC 7.3.0

When I run Pkg.test(“Flux”) I get the following:

Test Summary: | Pass Fail Error Total  
Flux | 448 4 3 455  
Throttle | 11 11  
Jacobian | 1 1  
Initialization | 12 12  
Params | 2 2  
Precision | 6 6  
Stacking | 3 3  
onecold | 4 4  
Optimise | 11 11  
Optimiser | 3 3  
Training Loop | 2 2  
basic | 17 17  
Dropout | 8 8  
BatchNorm | 13 13  
losses | 30 30  
Pooling | 2 2  
CNN | 1 1  
Depthwise Conv | 2 2  
Tracker | 261 261  
CuArrays | 8 8  
onecold gpu | 1 1  
CUDNN BatchNorm | 10 10  
RNN | 33 4 2 39  
R = Flux.RNN | 16 16  
R = Flux.GRU | 6 4 1 11  
batch\_size = 1 | 2 1 3  
batch\_size = 5 | 4 4 8  
R = Flux.LSTM | 11 1 12  
batch\_size = 1 | 9 9  
batch\_size = 5 | 2 1 3  
ERROR: LoadError: Some tests did not pass: 448 passed, 4 failed, 3 errored, 0 broken.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [March 26, 2019, 5:37pm UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003/2 "2019-03-26T17:37:35Z")

</div>

does this report any error?

```julia
(v1.1) pkg> build

```

---

<div class="post-metadata">

**Author:** ![bjenkinsgit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjenkinsgit/32/3440_2.png) [@bjenkinsgit](https://discourse.julialang.org/u/bjenkinsgit)\
**Post date:** [March 26, 2019, 7:41pm UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003/3 "2019-03-26T19:41:20Z")

</div>

I assume you mean: Pkg.build(“Flux”) ?

---

<div class="post-metadata">

**Author:** ![bjenkinsgit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjenkinsgit/32/3440_2.png) [@bjenkinsgit](https://discourse.julialang.org/u/bjenkinsgit)\
**Post date:** [March 26, 2019, 7:52pm UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003/4 "2019-03-26T19:52:00Z")

</div>

output from issuing the command Pkg.build(“Flux”)

julia\> Pkg.build(“Flux”)  
Building SpecialFunctions → `~/.julia/packages/SpecialFunctions/fvheQ/deps/build.log`  
Building ZipFile ─────────→ `~/.julia/packages/ZipFile/p60bh/deps/build.log`  
Building CodecZlib ───────→ `~/.julia/packages/CodecZlib/9jDi1/deps/build.log`

I get 0 errors. Re-running Pkg.test(“Flux”) gives me errors starting with GPU tests…

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [March 27, 2019, 12:07am UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003/5 "2019-03-27T00:07:57Z")

</div>

actually I meant `build` as that would build everything because I ran into issues with `arpack` doesn’t build but only affects GPU Flux.

---

<div class="post-metadata">

**Author:** ![bjenkinsgit](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bjenkinsgit/32/3440_2.png) [@bjenkinsgit](https://discourse.julialang.org/u/bjenkinsgit)\
**Post date:** [March 27, 2019, 1:57pm UTC](https://discourse.julialang.org/t/flux-train-failing-with-cuda-gpu/22003/6 "2019-03-27T13:57:30Z")

</div>

[SOLVED] Ok. The key step was to go into (what I can only assume is called) “command mode” by hitting the ‘]’ key (which gave me that “(v1.1) pkg” prompt and then typing “update”. This then updated my CUDA related packages and then built them for 1.1.0. Then, running “test Flux” gave me a clean and error free set of test results. Yay!