# Flux and cpu cores

**URL:** <https://discourse.julialang.org/t/flux-and-cpu-cores/45318>\
**Category:** Machine Learning\
**Tags:** parallel, multithreading, flux\
**Created:** [August 21, 2020, 10:53am UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318 "2020-08-21T10:53:12Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [August 21, 2020, 10:53am UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/1 "2020-08-21T10:53:12Z")

</div>

When I run the following code

```julia
using Flux
n = 100_000
p = 50
x = rand(Float32, p, n)
y = rand(Float32, n)    
trdata = Flux.Data.DataLoader(x, y, batchsize=100)
m = Chain(Dense(p, 100), Dense(100,100), Dense(100,1))
loss(x, y) = Flux.mse(m(x), y)
@time Flux.@epochs 10 Flux.train!(loss, Flux.params(m), trdata, Flux.ADAM())

```

on my laptop, all cores/threads are working at 100% which is somewhat surprising to me. Is this as expected? `Threads.nthreads()` returns 1. I use Julia 1.4.1 with Flux 0.10.4 on Ubuntu 18 with an Intel(R) Core™ i7-6600U (2 cores and 2 threads/core) CPU.

Suppose I want to train my model multiple times with different random initial weights, what would be the recommended way to do this?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [August 21, 2020, 11:52am UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/2 "2020-08-21T11:52:07Z")

</div>

It is because openblas is multi-threaded

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [August 21, 2020, 1:14pm UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/3 "2020-08-21T13:14:01Z")

</div>

Ok, thanks. Does this imply that I should avoid `Threads.@threads for ... end` in order to train several nets simultaneously?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [August 26, 2020, 2:33pm UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/4 "2020-08-26T14:33:15Z")

</div>

I think that if you use multi-threadding, it would automatically set the number of threads for OpenBlas to one. But in your case, it can still be a win.

---

<div class="post-metadata">

**Author:** ![Elrod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/elrod/32/22461_2.png) [@Elrod](https://discourse.julialang.org/u/Elrod)\
**Post date:** [August 26, 2020, 2:35pm UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/5 "2020-08-26T14:35:31Z")

</div>

> [@Tomas\_Pevny](#):
>
> I think that if you use multi-threadding, it would automatically set the number of threads for OpenBlas to one.

You [have to do this manually](https://discourse.julialang.org/t/why-doesnt-multithreading-help-here/45286/10). (When comparing the charts, note the different scales on the x-axis!)

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [August 27, 2020, 10:21am UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/6 "2020-08-27T10:21:34Z")

</div>

Thank you for the comments. I will make a basic comparison in a few days.

---

<div class="post-metadata">

**Author:** ![johnbb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnbb/32/34233_2.png) [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Post date:** [September 1, 2020, 9:29am UTC](https://discourse.julialang.org/t/flux-and-cpu-cores/45318/7 "2020-09-01T09:29:57Z")

</div>

I ran a few tests which confirm that `BLAS.set_num_threads(1)` should be set. On my system the code ran ~10 times faster. Please see below for codes and results

```julia
# start Julia with JULIA_NUM_THREADS=1 julia
using Flux
using BenchmarkTools
using LinearAlgebra
n = 100_000
p = 50
x = rand(Float32, p, n)
y = rand(Float32, n)    
trdata = Flux.Data.DataLoader(x, y, batchsize=100)
m = [Chain(Dense(p, 100), Dense(100,100), Dense(100,1)) for i in 1:4]
@btime for i in 1:4
    loss(x, y) = Flux.mse(m[i](x), y)
    Flux.@epochs 1 Flux.train!(loss, Flux.params(m[i]), trdata, Flux.ADAM())
end
# 6.286 s (1992500 allocations: 2.24 GiB)

# start Julia with JULIA_NUM_THREADS=4 julia
using Flux
using BenchmarkTools
using LinearAlgebra
n = 100_000
p = 50
x = rand(Float32, p, n)
y = rand(Float32, n)    
trdata = Flux.Data.DataLoader(x, y, batchsize=100)
m = [Chain(Dense(p, 100), Dense(100,100), Dense(100,1)) for i in 1:4]
@btime Threads.@threads for i in 1:4
    loss(x, y) = Flux.mse(m[i](x), y)
    Flux.@epochs 1 Flux.train!(loss, Flux.params(m[i]), trdata, Flux.ADAM())
end
# 10.864 s (1992523 allocations: 2.24 GiB)  

# start Julia with JULIA_NUM_THREADS=4 julia
using Flux
using BenchmarkTools
using LinearAlgebra
BLAS.set_num_threads(1)
n = 100_000
p = 50
x = rand(Float32, p, n)
y = rand(Float32, n)    
trdata = Flux.Data.DataLoader(x, y, batchsize=100)
m = [Chain(Dense(p, 100), Dense(100,100), Dense(100,1)) for i in 1:4]
@btime Threads.@threads for i in 1:4
    loss(x, y) = Flux.mse(m[i](x), y)
    Flux.@epochs 1 Flux.train!(loss, Flux.params(m[i]), trdata, Flux.ADAM())
end
# 1.076 s (1992515 allocations: 2.24 GiB)

```