# Flux: reproducibility of GPU experiments

**URL:** <https://discourse.julialang.org/t/flux-reproducibility-of-gpu-experiments/62092>\
**Category:** Machine Learning\
**Tags:** flux, cudnn, reproducibility\
**Created:** [May 30, 2021, 2:01pm UTC](https://discourse.julialang.org/t/flux-reproducibility-of-gpu-experiments/62092 "2021-05-30T14:01:26Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [May 30, 2021, 2:01pm UTC](https://discourse.julialang.org/t/flux-reproducibility-of-gpu-experiments/62092/1 "2021-05-30T14:01:26Z")

</div>

When I run the Flux model-zoo’s [conv\_mnist.jl](https://github.com/FluxML/model-zoo/blob/master/vision/conv_mnist/conv_mnist.jl) example there is an argument “seed” which should assure reproducibility (when larger than 0 ). This seems to only work when performing experiments on CPU, whereas GPU experiments give slightly different results.

When running on GPU epoch 0 losses and accuracies are identical between runs. However, as training progresses the losses evolve slightly differently between runs. It seems somehow the training process is non-deterministic when running on GPU. I am very curious if Flux has a way to get reproducible GPU results?

I googled around and found that something similar happens in pytorch because [cudnn may choose different implementations of algorithms between runs](https://pytorch.org/docs/stable/notes/randomness.html). Running “torch.use\_deterministic\_algorithms(True)” removes this source of indeterminism in pytorch. Is there a similar fix in Flux?

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [May 30, 2021, 5:41pm UTC](https://discourse.julialang.org/t/flux-reproducibility-of-gpu-experiments/62092/2 "2021-05-30T17:41:00Z")

</div>

As I understand it, most of the non-determinism comes from cuDNN. I’ve opened [Option for filtering by CUDNN\_DETERMINISTIC in `cudnnConvolutionAlgoPerfChoose` · Issue #938 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/issues/938) to track adding the right machinery to force it to be deterministic.