Hi, I’ve tried finding resources regarding Flux/Lux data parallelism. Im still unsure what the current SOTA solution for data parallel training would be. Im creating a different kinds of autoencoders, with the main one being a LSTM VAE, and due to the amount of data I want to process, dataparallelism would be great.
1 Like
Lux has it built in Distributed Data Parallel Training | Lux.jl Documentation. I had an older package now deprecated which still works with Flux GitHub - avik-pal/FluxMPI.jl: Distributed Data Parallel Training of Deep Neural Networks
2 Likes
Is this also the recommended way forward for single node data parallelism? Is there an alternative for Flux?