# Horovod and Flux / Julia
**URL:** https://discourse.julialang.org/t/horovod-and-flux-julia/53550
**Category:** Machine Learning
**Created:** [January 18, 2021, 5:28pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550 "2021-01-18T17:28:59Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [January 18, 2021, 5:28pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/1 "2021-01-18T17:28:59Z")
Hi all,
I wonder if anyone was trying to use horovod with Flux (or Julia in general).
Thanks for a answer.
Best wishes,
Tomas
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [January 30, 2021, 7:28pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/2 "2021-01-30T19:28:54Z")
I had a look through and Horovod is a C++ library under the hood, no? Unless somebody has wrapped it with a C API, I imagine the only straightforward way to use it would be via PyCall (if indeed that jives with all the `Distributed` machinery). AFAICT the only collective communication library that does expose a C interface would be NCCL. It would be nice to have something like this though, I’d rather not bother with getting CUDA-aware MPI working on my local machines.
---
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [January 31, 2021, 5:49am UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/3 "2021-01-31T05:49:48Z")
Thanks,
wrapping c++ in c seems to be doable, pycall sshould be easy.
---
### Author:  [@Palli](https://discourse.julialang.org/u/Palli)
#### Post date: [August 13, 2021, 4:49pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/4 "2021-08-13T16:49:33Z")
Why Hovorod? It seems to me DeepSpeed (and DeeperSpeed fork of it) might be for a similar purpose, and I think it might be better. I’m might be wrong but at least it has intriguing 1-bit LAMB (and before 1-bit Adam) breakthroughs:
[https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/](https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/)
I just think if anyone is planning to support these tools, start with the best one. Are there more similar?
> **[DeepSpeed Vs Horovod: A Comparative Analysis](https://analyticsindiamag.com/deepspeed-vs-horovod-a-comparative-analysis/)**
>
> A comparative analysis of open-source deep learning optimization libraries DeepSpeed and Horovod for advancing large-scale model training.
> DeepSpeed addresses the underlying performance difficulties and improves the speed and scale of the training with only a few lines of code change to the PyTorch model.
---
### Author:  [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)
#### Post date: [August 13, 2021, 7:02pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/5 "2021-08-13T19:02:27Z")
Agreed, one should go for the best.
I was asking at that time, as my former colleague was promoting to use horovod, but it never went beyond talks. We never moved to action. At the moment, many other things have piled up and I would not have time to do this.
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [August 15, 2021, 4:14pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/6 "2021-08-15T16:14:19Z")
Deepspeed is a C++ extension for a Python frontend. Horovod is a C++ library with a Python wrapper. Neither is terribly relevant for distributed training in Julia at the moment. What would be relevant is trying to port over some of the techniques used in ZeRO-{1,2,3,offload}, either in [https://github.com/DhairyaLGandhi/DaggerFlux.jl](https://github.com/DhairyaLGandhi/DaggerFlux.jl) or some alternative library.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [August 15, 2021, 4:23pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/7 "2021-08-15T16:23:20Z")
Why not FastAI?
[https://github.com/FluxML/FastAI.jl](https://github.com/FluxML/FastAI.jl)
---
### Author:  [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)
#### Post date: [August 15, 2021, 4:46pm UTC](https://discourse.julialang.org/t/horovod-and-flux-julia/53550/8 "2021-08-15T16:46:14Z")
The high-level abstractions in FastAI.jl are almost completely orthogonal to how distributed training is conducted. What will most likely happen is that FastAI uses the distributed functionality when it stabilizes.