# Implement berHu loss in Flux

**URL:** <https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707>\
**Category:** Machine Learning\
**Tags:** flux\
**Created:** [December 1, 2019, 4:17am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707 "2019-12-01T04:17:54Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![jchen975](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jchen975/32/8994_2.png) [@jchen975](https://discourse.julialang.org/u/jchen975)\
**Post date:** [December 1, 2019, 4:17am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/1 "2019-12-01T04:17:54Z")

</div>

Hi, I was trying to build this loss function from [this paper](https://arxiv.org/pdf/1606.00373.pdf):

![berhu](https://global.discourse-cdn.com/julialang/original/3X/3/2/32c86d28de579db5b6cd46550874e65c40b96795.png)

I wanted to see if it outperforms squared L2 norm in another regression problem I’m working on. Here is my attempt:

```julia
function berhu(x, y)
        x = model(x)
        loss = Tracker.collect(zeros(Float32, size(x)))
        bound = 0.2*maximum(abs.(x-y))
        inbound = abs.(x-y) .<= bound
        loss[inbound] .= norm.((x-y)[inbound], 1)
        loss[.!inbound] .= (((x-y)[.!inbound]).^2 .+ bound^2)./(2*bound)
        return loss
end

```

It works as intended when I comment out the `x = model(x)` line, with dummy variables. The problem when I run this is with `loss[inbound]`, which was brought up in issue [#93](https://github.com/FluxML/Flux.jl/issues/93) on Flux repo.

Stack trace of error:

```julia
ERROR: Can't differentiate `setindex!`
Stacktrace:
 [1] #setindex!#369(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::TrackedArray{…,SubArray{Float32,1,Array{Float32,1},Tuple{Array{Int64,1}},false}}, ::Float64, ::Int64) at C:\Users\me\.julia\packages\Tracker\JhqMQ\src\lib\array.jl:65
 [2] setindex!(::TrackedArray{…,SubArray{Float32,1,Array{Float32,1},Tuple{Array{Int64,1}},false}}, ::Float64, ::Int64) at C:\Users\me\.julia\packages\Tracker\JhqMQ\src\lib\array.jl:65
 [3] macro expansion at .\broadcast.jl:843 [inlined]
 [4] macro expansion at .\simdloop.jl:73 [inlined]
 [5] copyto! at .\broadcast.jl:842 [inlined]
 [6] copyto! at .\broadcast.jl:797 [inlined]
 [7] materialize!(::TrackedArray{…,SubArray{Float32,1,Array{Float32,1},Tuple{Array{Int64,1}},false}}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Nothing,typeof(norm),Tuple{Array{Float64,1},Int64}}) at .\broadcast.jl:756
 [8] top-level scope at none:0

```

How can I implement this loss function correctly?

---

<div class="post-metadata">

**Author:** ![Ratingulate](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ratingulate/32/9242_2.png) [@Ratingulate](https://discourse.julialang.org/u/Ratingulate)\
**Post date:** [December 1, 2019, 4:36am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/2 "2019-12-01T04:36:09Z")

</div>

Can you try on version 0.10 of flux?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 1, 2019, 4:50am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/3 "2019-12-01T04:50:19Z")

</div>

> [@jchen975](#):
>
> loss[inbound] .=

This is not going to work for any version of Flux. You have to rewrite it so that you do not mutate any vector

---

<div class="post-metadata">

**Author:** ![jchen975](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jchen975/32/8994_2.png) [@jchen975](https://discourse.julialang.org/u/jchen975)\
**Post date:** [December 1, 2019, 4:53am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/4 "2019-12-01T04:53:20Z")

</div>

Makes sense, but how do I implement it without mutating any vector?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 1, 2019, 4:56am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/5 "2019-12-01T04:56:58Z")

</div>

You can use boolean vectors that act like masks to blend the two together. Not sure if this is the best strategy though. Another alternative is to use mutation and define the gradient manually.

---

<div class="post-metadata">

**Author:** ![jchen975](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jchen975/32/8994_2.png) [@jchen975](https://discourse.julialang.org/u/jchen975)\
**Post date:** [December 1, 2019, 5:00am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/6 "2019-12-01T05:00:08Z")

</div>

Thanks, but I thought the `inbound = abs.(x-y) .<= bound` line is a boolean mask?

Also I just realized LossFunctions.jl is a thing, so I’ll take a look at its implementation for huber loss tomorrow.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 1, 2019, 5:01am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/7 "2019-12-01T05:01:24Z")

</div>

It is, but you use it for indexing rather than multiplication

---

<div class="post-metadata">

**Author:** ![jchen975](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jchen975/32/8994_2.png) [@jchen975](https://discourse.julialang.org/u/jchen975)\
**Post date:** [February 1, 2020, 12:24am UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/8 "2020-02-01T00:24:42Z")

</div>

It’s been a while, but any more suggestions? Thanks!  
[https://github.com/FluxML/Flux.jl/issues/1011](https://github.com/FluxML/Flux.jl/issues/1011)

---

<div class="post-metadata">

**Author:** ![DrChainsaw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drchainsaw/32/8497_2.png) [@DrChainsaw](https://discourse.julialang.org/u/DrChainsaw)\
**Post date:** [February 1, 2020, 4:19pm UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/9 "2020-02-01T16:19:06Z")

</div>

It seems like le\_float (i.e \<= for floats) is not differentiable. Perhaps you can use strictly less than (ie. \<) without any noticeable loss in performance or just reverse the in\_bounds to out\_bounds and use in\_bounds = .!out\_bounds.

---

<div class="post-metadata">

**Author:** ![jchen975](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jchen975/32/8994_2.png) [@jchen975](https://discourse.julialang.org/u/jchen975)\
**Post date:** [February 1, 2020, 9:06pm UTC](https://discourse.julialang.org/t/implement-berhu-loss-in-flux/31707/10 "2020-02-01T21:06:06Z")

</div>

Can’t believe I didn’t think of just using `.<` insteda of `.<=` haha, but it appears to be working now, thanks!