Let’s say I need my weights to be determined with an accuracy of 10^10 and hence want to train my network using Float64. Does it make sense to train my network with Float32 and then (after I get around 10^-7 accuracy) shift to Float64? Also, I have a feeling that this idea of first training w…

A possible way to improve training in Flux?

ChrisRackauckas August 12, 2020, 7:44am 16

Oh sorry, forgot why I had this tab open . Yes, changing precision in Julia is a fairly trivial Float32.(x) so abuse it to make multi-precision algorithms. Nick Higham gives a great talk on it:

2 Likes

Topic		Replies	Views
Bug? Using Flux & getting Float32 response on 64 bit Ubuntu OS Machine Learning	4	1242	March 4, 2019
Will decreasing the precision of intermediate variables improve performance of code? Performance performance	12	1233	May 9, 2020
Why does this Flux code fail with Float64, but run for Float32? New to Julia flux	0	595	November 7, 2021
Generic way to change float precision in FluxML Machine Learning flux , fastai	3	234	August 18, 2023
The same network performs differently in Flux.jl and tensorflow Machine Learning performance	13	3122	December 18, 2019

A possible way to improve training in Flux?

Related topics