# How can I make Flux use all my CPUs?

**URL:** <https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869>\
**Category:** General Usage\
**Created:** [March 14, 2019, 2:09pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869 "2019-03-14T14:09:54Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![yao](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@yao](https://discourse.julialang.org/u/yao)\
**Post date:** [March 14, 2019, 2:09pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/1 "2019-03-14T14:09:54Z")

</div>

Hi all,

I already set `export JULIA_NUM_THREADS=12` and in REPL I checked `Threads.nthreads()` =12. But when I ran the training, I only saw Julia(Flux) using 8 CPUs, and, they were not in full power.

How can I make Flux use all my CPUs and run in full power?

Thanks!

---

<div class="post-metadata">

**Author:** ![uadjet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uadjet/32/4041_2.png) [@uadjet](https://discourse.julialang.org/u/uadjet)\
**Post date:** [March 14, 2019, 2:15pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/2 "2019-03-14T14:15:53Z")

</div>

Not sure if this is relevant to your specific case, but this happens to me if my batch\_size/sample\_count aren’t tuned correctly; usually because the CPU is spending all of it’s time moving things around as opposed to actually computing.

---

<div class="post-metadata">

**Author:** ![yao](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@yao](https://discourse.julialang.org/u/yao)\
**Post date:** [March 14, 2019, 2:33pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/3 "2019-03-14T14:33:56Z")

</div>

Thank you and any guidance on how to tune it correctly?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [March 14, 2019, 3:33pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/4 "2019-03-14T15:33:00Z")

</div>

Try to increase number of threads that openblas (OPENBLAS\_NUM\_THREADS) uses, since at the moment Flux does not have any paralelization support.

---

<div class="post-metadata">

**Author:** ![uadjet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/uadjet/32/4041_2.png) [@uadjet](https://discourse.julialang.org/u/uadjet)\
**Post date:** [March 14, 2019, 3:46pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/5 "2019-03-14T15:46:49Z")

</div>

This is actually a point on which I don’t know if there is a ‘correct’ answer. Choosing batch and sample sizes is usually a bit of trial and error for me. I do have a heuristic that I try to stick to and it has served me well for my purposes. I do tend to use a GPU for my work, so take this with a grain of salt in that regard.

\text{Weight size (bits)}\ \cdot\ \text{#of weights in your model} \ \cdot\ nsamples\ \cdot\ batchsize\ \approx\ \text{Size of Memory}

With Flux, it seems that most of the defaults are Float32’s for ease of computing with GPUs, meaning your Weight Size parameter is 32bits.

If anyone has a better system, I’d be really curious.

Also @Tomas_Pevny is almost certainly correct with the solution to your initial problem but I didn’t want to leave you hanging on your followup.

---

<div class="post-metadata">

**Author:** ![yao](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@yao](https://discourse.julialang.org/u/yao)\
**Post date:** [March 15, 2019, 7:46am UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/6 "2019-03-15T07:46:23Z")

</div>

Thank you both @Tomas_Pevny @uadjet . But I am afraid that Flux does not support multi-cpu parallelization.  
I set both

```julia
JULIA_NUM_THREADS=12
OPENBLAS_NUM_THREADS=12

```

and I change the batch size.  
Still Flux does not use 12 CPUs, and, what’s worse, single CPU sometimes.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [March 15, 2019, 1:41pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/7 "2019-03-15T13:41:58Z")

</div>

Correct, _Flux_ does not itself support multi-CPU parallelism (yet). However, because it uses OpenBLAS for its BLAS operations (like the rest of Julia), and because OpenBLAS _does_ do multithreading _sometimes_, then you _sometimes_ see multiple CPUs getting used. Whether all 12 of them will get used is up to what BLAS routines are called, and what OpenBLAS thinks is efficient (sometimes adding more CPUs might not help compute something faster, or can even slow things down).

So the bottom line is that you shouldn’t always expect Flux to use 100% of all your available CPUs; for that to be the case requires the right set of circumstances and the “right” calls into BLAS.

---

<div class="post-metadata">

**Author:** ![yao](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@yao](https://discourse.julialang.org/u/yao)\
**Post date:** [March 15, 2019, 1:50pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/8 "2019-03-15T13:50:10Z")

</div>

@jpsamaroo The problem is I want to run my program in the workstation and Flux does not use all of the CPU power. I found that when I ran Flux on a 64-core machine, it used only 8 of them, making the training process very very slow.  
I also can only use Flux for very small data sets.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [March 15, 2019, 2:07pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/9 "2019-03-15T14:07:00Z")

</div>

Any chance you can either share your code here, or run your code under profiling (`Profile.@profile`)? That way we can start to investigate where your bottlenecks are and determine if there’s anything that can be done to improve the CPU utilization.

---

<div class="post-metadata">

**Author:** ![yao](https://avatars.discourse-cdn.com/v4/letter/y/2bfe46/32.png) [@yao](https://discourse.julialang.org/u/yao)\
**Post date:** [March 15, 2019, 3:26pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/10 "2019-03-15T15:26:03Z")

</div>

@jpsamaroo Sure. That would be great help. It’s a very simple regression problem:  
y=f(x\_1,x\_2,x\_3,x\_4,x\_5,x\_6)  
The code is simple too:

```julia
using Flux
using Flux: throttle
using Base.Iterators: repeated
using Pkg
using DelimitedFiles

X_train = readdlm("X_train.txt", ' ', Float64)
y_train = readdlm("y_train.txt", ' ', Float64)
X_valid = readdlm("X_valid.txt", ' ', Float64)
y_valid = readdlm("y_valid.txt", ' ', Float64)

X_train = transpose(X_train)
y_train = transpose(y_train)

X_valid = transpose(X_valid)
y_valid = transpose(y_valid)

dataset = repeated((X_train, y_train), 5000)

m = Chain(Dense(6, 256, relu),
                BatchNorm(256, relu),
                Dense(256, 256, relu),
                BatchNorm(256, relu),
                Dense(256, 1, relu))
println(m)
loss(x, y) = Flux.mse(m(x), y)
evalcb = () -> @show(loss(X_valid, y_valid))
opt = ADAM(0.02)
Flux.train!(loss, params(m), dataset, opt, cb = throttle(evalcb, 10))

```

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [March 15, 2019, 7:58pm UTC](https://discourse.julialang.org/t/how-can-i-make-flux-use-all-my-cpus/21869/11 "2019-03-15T19:58:02Z")

</div>

I didn’t know this before, but apparently Julia’s OpenBLAS is hard-capped to 16 threads (which may show up as 800% CPU on your system): [OpenBLAS: Julia slower than R - #17 by Elrod](https://discourse.julialang.org/t/julia-slower-than-r/21594/17)