# Flux training GPU vs CPU different results
**URL:** https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165
**Category:** Machine Learning
**Tags:** gpu, flux
**Created:** [August 18, 2020, 3:40pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165 "2020-08-18T15:40:40Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@luboshanus](https://discourse.julialang.org/u/luboshanus)
#### Post date: [August 18, 2020, 3:40pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/1 "2020-08-18T15:40:40Z")
Hi,
I have a problem while training on GPU vs CPU. Or when I just once run train function it goes completely off while on GPU. If I choose CPU it just stays “converge” and not go further away. An example of 3 runs on GPU. It just gets super off, and I don’t get that when doing function `train2()` again it starts close where the previous ended. It just should not because I guess the variables are not GLOBAL.
When I run it as many times I want, it will always give a result similar to the first one since it just starts with different values but then converges somewhere. GPU just goes off.
```julia
julia> m, argsout, resout = train2(tr_data, te_data, modelfct;...);
[ Info: Training on GPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #879)
[ Info: Start Training . . .
Epoch: 0 Train: 1.0487 Test: 43.2849
Epoch: 30 Train: 1.1291 Test: 1.1006
Epoch: 60 Train: 1.1291 Test: 1.1006
Epoch: 90 Train: 1.1291 Test: 1.1006
Epoch: 120 Train: 1.1291 Test: 1.1006
Epoch: 150 Train: 1.1291 Test: 1.1006
julia> m, argsout, resout = train2(tr_data, te_data, modelfct;...);
[ Info: Training on GPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #879)
[ Info: Start Training . . .
Epoch: 0 Train: 1.1291 Test: 1.1006
Epoch: 30 Train: 1.1291 Test: 1.1006
Epoch: 60 Train: 1.1291 Test: 1.1006
Epoch: 90 Train: 1.1291 Test: 1.1006
Epoch: 120 Train: 131.7943 Test: 169.8546
Epoch: 150 Train: 46.2814 Test: 97.1106
julia> m, argsout, resout = train2(tr_data, te_data, modelfct;...);
[ Info: Training on GPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #879)
[ Info: Start Training . . .
Epoch: 0 Train: 160.3743 Test: 143.1688
Epoch: 30 Train: 160.3743 Test: 152.41
Epoch: 60 Train: 116.2331 Test: 95.1814
Epoch: 90 Train: 107.7877 Test: 159.5695
Epoch: 120 Train: 105.0404 Test: 54.0994
Epoch: 150 Train: 99.2565 Test: 69.0809
```
Any ideas are more than welcomed.
---
### Author:  [@darsnack](https://discourse.julialang.org/u/darsnack)
#### Post date: [August 18, 2020, 4:04pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/2 "2020-08-18T16:04:07Z")
Based on the output, it definitely seems like something is being mutated. So each successive run is picking up where the previous run left off instead of starting from scratch. Do you mind posting the CPU version of the output you shared? From your description, it seems like the CPU version doesn’t have the mutation issue.
---
### Author:  [@luboshanus](https://discourse.julialang.org/u/luboshanus)
#### Post date: [August 19, 2020, 7:53am UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/3 "2020-08-19T07:53:32Z")
Hi, thanks. It looks like you write, but I just can’t find it. Here are 3 runs of CPU and 3 runs on GPU. CPU behaves as I would expect - learns a bit, has the same starting loss and does not go off some stable path.
```julia
julia> m, argsout, resout = train2(tr_data, te_data, modelfct; cuda=false,...);
[ Info: Training on CPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #185)
[ Info: Start Training . . .
Epoch: 0 Train: 1.1564 Test: 1.1672
Epoch: 30 Train: 1.0754 Test: 1.08
Epoch: 60 Train: 1.0749 Test: 1.0777
Epoch: 90 Train: 1.0767 Test: 1.0828
Epoch: 120 Train: 1.073 Test: 1.08
Epoch: 150 Train: 1.0771 Test: 1.0833
julia> m, argsout, resout = train2(tr_data, te_data, modelfct; cuda=false,...);
[ Info: Training on CPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #185)
[ Info: Start Training . . .
Epoch: 0 Train: 1.1564 Test: 1.1672
Epoch: 30 Train: 1.0754 Test: 1.08
Epoch: 60 Train: 1.0749 Test: 1.0777
Epoch: 90 Train: 1.0767 Test: 1.0828
Epoch: 120 Train: 1.073 Test: 1.08
Epoch: 150 Train: 1.0771 Test: 1.0833
julia> m, argsout, resout = train2(tr_data, te_data, modelfct; cuda=false,...);
[ Info: Training on CPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #185)
[ Info: Start Training . . .
Epoch: 0 Train: 1.1564 Test: 1.1672
Epoch: 30 Train: 1.0754 Test: 1.08
Epoch: 60 Train: 1.0749 Test: 1.0777
Epoch: 90 Train: 1.0767 Test: 1.0828
Epoch: 120 Train: 1.073 Test: 1.08
Epoch: 150 Train: 1.0771 Test: 1.0833
julia> m, argsout, resout = train2(tr_data, te_data, modelfct; cuda=true,...);
[ Info: Training on GPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #185)
[ Info: Start Training . . .
Epoch: 0 Train: 1.1589 Test: 1.2147
Epoch: 30 Train: 1.0768 Test: 1.0803
Epoch: 60 Train: 41.8823 Test: 41.5863
Epoch: 90 Train: 1.068 Test: 1.0709
Epoch: 120 Train: 37.0247 Test: 82.1282
Epoch: 150 Train: 44.2864 Test: 1.0749
julia> m, argsout, resout = train2(tr_data, te_data, modelfct; cuda=true,...);
[ Info: Training on GPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #185)
[ Info: Start Training . . .
Epoch: 0 Train: 34.6607 Test: 40.7039
Epoch: 30 Train: 43.6453 Test: 1.0864
Epoch: 60 Train: 12.0337 Test: 80.6082
Epoch: 90 Train: 48.6058 Test: 45.9999
Epoch: 120 Train: 22.2186 Test: 1.0751
Epoch: 150 Train: 22.3228 Test: 138.1419
julia> m, argsout, resout = train2(tr_data, te_data, modelfct; cuda=true,...);
[ Info: Training on GPU
[ Info: Dataset: 1600 train and 400 test examples
[ Info: Model: 1185 trainable params. Chain(Dense(1, 64, relu), Dense(64, 16, relu), Dropout(0.5), Dense(16, 1), #185)
[ Info: Start Training . . .
Epoch: 0 Train: 31.0082 Test: 50.5443
Epoch: 30 Train: 45.2076 Test: 49.8443
Epoch: 60 Train: 46.5327 Test: 1.0789
Epoch: 90 Train: 55.8642 Test: 122.2016
Epoch: 120 Train: 45.7922 Test: 46.6472
Epoch: 150 Train: 86.8372 Test: 47.0398
```
---
### Author:  [@luboshanus](https://discourse.julialang.org/u/luboshanus)
#### Post date: [August 20, 2020, 10:14am UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/4 "2020-08-20T10:14:02Z")
Hi, I would add a figure where I got from yesterday’s situation.
It works “better” but it is still super random the training on GPU. Code is just the same, data too, and I just say device is `gpu` or `cpu` and the losses look like on the figure attached. I just don’t understand.
Does anyone know what might be the problem? Thanks a lot!
---
### Author:  [@darsnack](https://discourse.julialang.org/u/darsnack)
#### Post date: [August 20, 2020, 12:42pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/5 "2020-08-20T12:42:21Z")
You’ll have to post the inside of `train2` since that’s where the mutation must be occurring. This seems like less of a ML issue and more of a programming error.
---
### Author:  [@luboshanus](https://discourse.julialang.org/u/luboshanus)
#### Post date: [August 20, 2020, 3:18pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/6 "2020-08-20T15:18:24Z")
I think that the problem was too many `Flux.reset!()` in `for loop` of `1:epochs` when doing RNN. I have rewritten the code from almost scratch inspired by `mlp.jl` from `model-zoo` and find out that `Flux.reset!()` is needed only in the loss function and when I do run model. It might be helpful for someone to write and to put the model into `let+end`, because I have wanted to save prediction when improved.
```julia
if condtion_that_model_improved
let m1 = cpu(model)
update_best_yhat = m1(test_data_input)
end
end
```
and continue learning further.
---
### Author:  [@darsnack](https://discourse.julialang.org/u/darsnack)
#### Post date: [August 20, 2020, 3:39pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/7 "2020-08-20T15:39:46Z")
I am glad you figured it out. Since RNNs have hidden state, doing inference passes in the middle of training could cause issues. The cleanest solution here would probably be to not use `Flux.train!` if you are and write your own training loop instead. This would allow you to have a copy of each forward pass which you can then store for future reference. For example,
```julia
besty = # initialize
for epoch in 1:nepochs
for (x, y) in data
local ypred
grad = gradient(params(m)) do
ypred = m(x)
end
update!(opt, params(m), grad)
if improved(ypred, besty)
besty = ypred
end
end
end
```
---
### Author:  [@german\_brunini](https://discourse.julialang.org/u/german_brunini)
#### Post date: [August 20, 2020, 8:19pm UTC](https://discourse.julialang.org/t/flux-training-gpu-vs-cpu-different-results/45165/8 "2020-08-20T20:19:02Z")
@darsnack I have a similar problem, but I believe is not related to GPU vs CPU. The problem appears when I wrap the training code inside a `function()` which acctually prevents me to upgrade the code to a more dynamic custom training loop. when I do the latter, it just gives me the same result allways, but when I run it from the REPL, it runs OK. I have made a different [post](https://discourse.julialang.org/t/training-gets-differents-results-when-using-flux-train-inside-function/45284) for this issue. I don’t know if it would be better to just continue this one… let me know @luboshanus if that’s Ok.