# Can Flux handle multiple GPUs?

**URL:** <https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323>\
**Category:** Machine Learning\
**Created:** [August 5, 2022, 3:45am UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323 "2022-08-05T03:45:25Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lewis\_Hein](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lewis_hein/32/23540_2.png) [@Lewis\_Hein](https://discourse.julialang.org/u/Lewis_Hein)\
**Post date:** [August 5, 2022, 3:45am UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/1 "2022-08-05T03:45:25Z")

</div>

I am playing around with ML in Julia and stumbled on a source of cheap nvidia Quadro K2000s. Far from modern, I know, but I can’t afford modern hardware for side projects. If I stuck 4 of them in 1 tower, would Flux/Cuda.jl take advantage of all 4?

Currently my models are training on a a single core of a 12-core machine because Flux.jl doesn’t do CPU parallelism. I’m sure any CUDA.jl compatible GPU would help a lot; would 4 of them help 4 times as much?

Many thanks! As always, I love Julia and everything it makes possible for me 😃

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 5, 2022, 4:11am UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/2 "2022-08-05T04:11:17Z")

</div>

> **[GitHub - avik-pal/FluxMPI.jl: Distributed Data Parallel Training of Deep...](https://github.com/avik-pal/FluxMPI.jl)**
>
> Distributed Data Parallel Training of Deep Neural Networks - GitHub - avik-pal/FluxMPI.jl: Distributed Data Parallel Training of Deep Neural Networks

see also [Multi-GPU single host training example - #2 by j\_u](https://discourse.julialang.org/t/multi-gpu-single-host-training-example/73689/2)

---

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 5, 2022, 12:32pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/3 "2022-08-05T12:32:29Z")

</div>

Hi,

Looking up the quadro k2000 it is a 9 year old card with 2gb vram. You can find it in the table [CUDA-Enabled NVIDIA Quadro and NVIDIA RTX](https://developer.nvidia.com/cuda-gpus) on this page where it is classified as having “compute capability 3.0”. This means that a lot of deep learning frameworks wont support it, and even if they do it will be very slow. Having 4 of these will probably also introduce additional overhead related to piping data and accumulating gradients.

But the good news is that Flux is capable of using all physical cores:) Perhaps you need to set the JULIA\_NUM\_THREADS environmental variable. What output do you get when running Threads.nthreads() in the REPL?

---

<div class="post-metadata">

**Author:** ![Lewis\_Hein](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lewis_hein/32/23540_2.png) [@Lewis\_Hein](https://discourse.julialang.org/u/Lewis_Hein)\
**Post date:** [August 5, 2022, 1:34pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/4 "2022-08-05T13:34:40Z")

</div>

![image](https://global.discourse-cdn.com/julialang/original/3X/8/6/8676e72fb4adf279b8c28942bcf6a399fec510ba.png)  
(12 cores w/hyperthreading)

Julia CPU usage never rose above 5%. I never saw any use of \>1 core while the model was training. I tried installing Octavian, then restarting the notebook and reloading packages. Nothing I did made any difference.

Clearly I am missing something. What is it?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 5, 2022, 1:43pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/5 "2022-08-05T13:43:15Z")

</div>

> [@Lewis\_Hein](#):
>
> I never saw any use of \>1 core while the model was training. I tried installing Octavian, then restarting the notebook and reloading packages. Nothing I did made any difference.

Julia doessn’t do automatic threading, it’s all up to Flux I guess

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 5, 2022, 2:14pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/6 "2022-08-05T14:14:24Z")

</div>

I would also take a closer look at specification and software requirements in case of older GPU cards. In case of training on CPUs, my experience suggests that `Julia Distributed` capabilities could be more useful than multithreading. Once I attempted to train quite demanding deep learning model only on CPUs. Here is some info: [link](https://github.com/jonathan-laurent/AlphaZero.jl/issues/71) Disclaimer is that this was almost my very first contact with Julia so not everything might be correct but in general it should not be that bad. It depends how you look but I am recalling that the first, major part of the training took about the same time on high end CPUs (`2 x Platinum 8358`) as on high end GPU (`V100`). [I wrote it but due to some reasons it is a very simplified comparison, there were some GPU inefficiencies and probably the cost of the CPUs, board and RAM exceeded the cost of high end GPU]. I personally never tried `FluxMPI.jl` but my attempts with bare `MPI.jl` were not so great but it was probably the case of the model I used and my basic knowledge about `MPI`. There was a very interesting presentation at the last JuliaCon “[Scaling up Training of Any Flux.jl Model Made Easy | Dhairya Gandhi | JuliaCon 2022](https://www.youtube.com/watch?v=176c4S6LqT8)” and as for `DaggerFlux.jl` and `Dagger.jl` I believe should you pursue this road you will receive excellent support especially at the public forums. As for directed acyclic graphs you may also take a look at the very interesting “[Introduction to Graph Computing | JuliaCon 2022 | Yadong Li](https://www.youtube.com/watch?v=pxs7ysnb15A)”

I believe that the idea of CPU training is quite valid especially in case of large models, cloud environments and discounted pricing and should be even more realistic with the next CPU generation. However, in general this is a contrary view and probably a little futuristic. I have seen several threads on this topic recently and I have tried to discuss this subject several times by myself however as I wrote this (ML/DL on CPUs) seems to be a niche topic and in most cases, should not be probably a favored option currently.

---

<div class="post-metadata">

**Author:** ![Lewis\_Hein](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lewis_hein/32/23540_2.png) [@Lewis\_Hein](https://discourse.julialang.org/u/Lewis_Hein)\
**Post date:** [August 5, 2022, 2:16pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/7 "2022-08-05T14:16:28Z")

</div>

So, how do I get Flux to use multiple CPU threads while training? Apparently there is a way, but it’s not shown in any docs that I have found. Do I have to write some kind of custom training loop and decorate it with @parallel? Is there a way to tell Flux.train!() that I want it to be multithreaded?

I am super new to the Julia ML ecosystem so still feeling my way forward in the dark lol

---

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 5, 2022, 2:21pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/8 "2022-08-05T14:21:28Z")

</div>

That is strange. The kernels Flux uses for matmul and conv should use multiple threads by default. And that is the behaviour I observe. Here is a screenshot of CPU utilization (htop) on my 8-core laptop while training the [conv\_mnist example from the model zoo](https://github.com/FluxML/model-zoo/tree/master/vision/conv_mnist). I saw similar CPU utilization when running the mnist\_mlp example.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/2/d24dd51fe7efe6c4421fe7badc22050803dc9342.png)

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 5, 2022, 2:26pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/9 "2022-08-05T14:26:02Z")

</div>

> [@Lewis\_Hein](#):
>
> how do I get Flux to use multiple CPU threads while training?

Please take a look at the links I provided above. There are some code snippets related to `Distributed` training of `AlphaZero` (in this case its more about processes than threads). Hope you may find something useful there. And again, I would especially recommend looking at the provided videos, as just to quote The Mandalorian, “This is the way”. (I mean directed acyclic graphs could become the predominant method in the future). :- ) EDIT: From my side, I would like to add that on `Julia` below `1.8` there is a caveat related to `BLAS` settings. I mean `Julia` starts by default with max 8 `BLAS` threads and almost whatever you do you can not go above 32. With 1.8 this restriction was removed.

---

<div class="post-metadata">

**Author:** ![Lewis\_Hein](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lewis_hein/32/23540_2.png) [@Lewis\_Hein](https://discourse.julialang.org/u/Lewis_Hein)\
**Post date:** [August 5, 2022, 2:52pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/10 "2022-08-05T14:52:39Z")

</div>

That model from the model zoo started off like this:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/8/6864b294d8d163c63e318f2ea885e5c8775c7a8a.png)

I am guessing this was some kind of compilation step.

Now it looks more like this:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/3/7/37e0c03746ffd0845bd9406d9d7c73b1b207375b.png)

I am going to have to dig through the script and see what I did wrong in my code

---

<div class="post-metadata">

**Author:** ![Lewis\_Hein](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lewis_hein/32/23540_2.png) [@Lewis\_Hein](https://discourse.julialang.org/u/Lewis_Hein)\
**Post date:** [August 5, 2022, 2:53pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/11 "2022-08-05T14:53:38Z")

</div>

I will do that, and probably come back with lots of new questions 😃

---

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 5, 2022, 3:03pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/12 "2022-08-05T15:03:22Z")

</div>

That looks like it is using 8 threads 🙂  
Dense layers mostly rely on matrix multiplication which is done using BLAS kernels.

the package LinearAlgebra has the functions BLAS.get\_num\_threads and BLAS.set\_num\_threads, which you can use to change how many threads BLAS uses.

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 5, 2022, 3:05pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/13 "2022-08-05T15:05:55Z")

</div>

You are always welcomed. I will be delighted to learn more about ML/DL. At the same time please be advised that I am not a long time Julia user, nor coding is my area of expertise. However, I would like to reiterate that as for `Dagger` and `DaggerFlux` I am pretty sure you receive a top notch support at public forums directly from the maintainers.

---

<div class="post-metadata">

**Author:** ![Lewis\_Hein](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lewis_hein/32/23540_2.png) [@Lewis\_Hein](https://discourse.julialang.org/u/Lewis_Hein)\
**Post date:** [August 5, 2022, 3:07pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/14 "2022-08-05T15:07:57Z")

</div>

Exactly. 8 is progress! I seem to remember that BLAS is limited to 8 threads below Julia 1.8, right?

---

<div class="post-metadata">

**Author:** ![Rasmus\_Hoier](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rasmus_hoier/32/24036_2.png) [@Rasmus\_Hoier](https://discourse.julialang.org/u/Rasmus_Hoier)\
**Post date:** [August 5, 2022, 3:11pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/15 "2022-08-05T15:11:56Z")

</div>

I don’t think so. I am pretty sure I tried using 16 threads on a 16 core machine a while ago on either Julia 1.5 or julia 1.6. But I don’t think runtime decreased much for me even though all threads were busy.

---

<div class="post-metadata">

**Author:** ![j\_u](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/j_u/32/219081_2.png) [@j\_u](https://discourse.julialang.org/u/j_u)\
**Post date:** [August 5, 2022, 3:20pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/16 "2022-08-05T15:20:52Z")

</div>

Its hard to refer without exact example however re 8 BLAS threads pls see: [BLAS performance testing for Julia 1.8](https://discourse.julialang.org/t/blas-performance-testing-for-julia-1-8/69520). Not sure if this could be to iterest, however, one may also easily change BLAS backend thanks to `libblastrampoline` and instead of default `OpenBLAS` use other libraries like i.e. `MKL`. Depending on the matrix size and the hardware sometimes this can provide significant benefits. `BLAS` performance can be checked with i.e. `BLASBenchmarksCPU.jl` or `BLASBenchmarksGPU.jl`. Hope it helps.

---

<div class="post-metadata">

**Author:** ![ToucheSir](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/touchesir/32/14411_2.png) [@ToucheSir](https://discourse.julialang.org/u/ToucheSir)\
**Post date:** [August 5, 2022, 10:23pm UTC](https://discourse.julialang.org/t/can-flux-handle-multiple-gpus/85323/17 "2022-08-05T22:23:09Z")

</div>

You can expect a minimum of 10s of compilation latency when taking the first gradient of Flux models (this 10s comes from the AD, not Flux) and possibly more depending on model complexity (\>1min is not unheard of for complex vision models, some of this does come from Flux). One way to evaluate _only_ runtime is to call `gradient` with your loss, model and some dummy data first and then do your perf measurements.