# Flux example 100x slower with L2 regularisation(?!)

**URL:** <https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914>\
**Category:** Performance\
**Tags:** question\
**Created:** [November 10, 2020, 1:53pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914 "2020-11-10T13:53:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![compleat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/compleat/32/8958_2.png) [@compleat](https://discourse.julialang.org/u/compleat)\
**Post date:** [November 10, 2020, 1:53pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914/1 "2020-11-10T13:53:14Z")

</div>

I am running a network in Flux/ADAM without regularisation which does 100 epochs in about as many seconds.

When I add an L2-penalty of 0.01\*sum(abs2 , m[2].W) to the loss function and train, it takes 90-100 seconds per epoch on exactly the same data and network.

Can anyone think of a reason this might be happening, and how I might go about fixing it?

I am running a 100x50x50x1 MLP on 8 cores (linux)

Thanks for any thoughts.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [November 10, 2020, 2:18pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914/2 "2020-11-10T14:18:21Z")

</div>

> [@compleat](#):
>
> 0.01\*sum(abs2 , m[2].W)

Maybe the `0.01` makes things promote to `Float64` and some slower fallback is used? But if you are running on a CPU, that might be unlikely.

---

<div class="post-metadata">

**Author:** ![compleat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/compleat/32/8958_2.png) [@compleat](https://discourse.julialang.org/u/compleat)\
**Post date:** [November 10, 2020, 2:20pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914/3 "2020-11-10T14:20:32Z")

</div>

Thanks. I will and will let you know. Is there another way to set \lambda?

---

<div class="post-metadata">

**Author:** ![compleat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/compleat/32/8958_2.png) [@compleat](https://discourse.julialang.org/u/compleat)\
**Post date:** [November 10, 2020, 2:27pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914/4 "2020-11-10T14:27:58Z")

</div>

That is the answer! I convered 0.01 to Float32 and it was off and running again!  
Thanks!

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [November 10, 2020, 2:42pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914/5 "2020-11-10T14:42:54Z")

</div>

This seems to have been worked on lately, BTW. Here’s a factor of 100 (CPU):

```julia
julia> @btime x*y setup=(x=randn(100,100); y=randn(100,100));
  9.248 μs (2 allocations: 78.20 KiB)

julia> @btime x*y setup=(x=randn(Float32, 100,100); y=randn(Float32, 100,100));
  6.198 μs (2 allocations: 39.14 KiB)

julia> @btime x*y setup=(x=randn(Float32, 100,100); y=randn(100,100));
  711.042 μs (8 allocations: 78.53 KiB)

julia> VERSION
v"1.3.0"

```

but on v1.5, it’s more like a factor of 3 for me, I guess it converts one array first.

---

<div class="post-metadata">

**Author:** ![compleat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/compleat/32/8958_2.png) [@compleat](https://discourse.julialang.org/u/compleat)\
**Post date:** [November 10, 2020, 2:51pm UTC](https://discourse.julialang.org/t/flux-example-100x-slower-with-l2-regularisation/49914/6 "2020-11-10T14:51:20Z")

</div>

Hi. I was running on 1.5.2