Mix-Precision training using Models of Metalhead (Flux.jl) or Boltz (Lux.jl)

It is possible to train Deep Learning models for computer vision available in packages like Metalhead (Flux) or Boltz (Lux) with mix-precision?

Mixed precision training of any model is possible, but you will have to write a lot of the logic such as the loss scaling yourself. Rule for mixed precision training by CarloLucibello · Pull Request #152 · FluxML/Optimisers.jl · GitHub may help with some of that.

Thanks for the tip!

1 Like

If do you come up with a full example of mixed precision training and can share it, please do! I think it would help with developing some utilities and abstractions around this going forwards.

1 Like