Mix-Precision training using Models of Metalhead (Flux.jl) or Boltz (Lux.jl)

Specific Domains Machine Learning
,
1

It is possible to train Deep Learning models for computer vision available in packages like Metalhead (Flux) or Boltz (Lux) with mix-precision?

2

Mixed precision training of any model is possible, but you will have to write a lot of the logic such as the loss scaling yourself. Rule for mixed precision training by CarloLucibello · Pull Request #152 · FluxML/Optimisers.jl · GitHub may help with some of that.

3

Thanks for the tip!

1 Like
4

If do you come up with a full example of mixed precision training and can share it, please do! I think it would help with developing some utilities and abstractions around this going forwards.

1 Like