# Request to upgrade to LossFunctions.jl

**URL:** <https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738>\
**Category:** Machine Learning\
**Tags:** package, release, machine-learning\
**Created:** [April 21, 2023, 12:06pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738 "2023-04-21T12:06:29Z")\
**Posts on this page:** 17\
**Page:** 2

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 24, 2023, 11:44pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/21 "2023-04-24T23:44:58Z")

</div>

Any suggestion on how to handle `deriv` and `deriv2` in this approach @aplavin ? Currently we can perform these aggregations with first and second derivatives as well.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 24, 2023, 11:46pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/22 "2023-04-24T23:46:52Z")

</div>

The design space here is a bit difficult to navigate, but I think we are converging into something that has all the benefits of nice Julian syntax, nice performance and clean implementation.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [April 24, 2023, 11:52pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/23 "2023-04-24T23:52:12Z")

</div>

> [@aplavin](#):
>
> I think the “implicit broadcasting” methods (like `L1DistLoss()([1,2,3], [2,5,-2])`) should just be removed

I think you’re losing sight of the actual use case for these loss functions. Within the context of MLJ, loss functions (called “measures” in MLJ) are used for evaluating machine learning models, with a typical use case looking like this:

```julia
evaluate!(
    mach,
    resampling = CV(nfolds=5),
    measure = [rms, mae]
)

```

With the approach you’re advocating, I would have to write that like this:

```julia
evaluate!(
    mach,
    resampling = CV(nfolds=5),
    measure = [
        (ŷ, y) -> sqrt(mean(L2DistLoss(), ŷ, y)),
        (ŷ, y) -> mean(L1DistLoss(), ŷ, y))
    ]
)

```

If we take your approach to its logical conclusion, no one would ever be allowed to write a function that takes an `AbstractArray` argument (unless that `AbstractArray` represents a mathematical vector, matrix, or tensor). But that’s a bit pedantic—factoring out commonly used expressions into separate functions is a normal part of programming.

All that being said, my feeling is that these metric functions are so trivial that there’s not much to be gained by attempting to get Flux and MLJ to use the same loss function implementation. (Is that the implicit goal of LossFunctions.jl?)

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 24, 2023, 11:55pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/24 "2023-04-24T23:55:04Z")

</div>

You are quoting me on implicit broadcasting methods, but your example is on the aggregation. These two are effectively independent. Maybe the latter should stay, idk, but the former should really go away.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 24, 2023, 11:55pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/25 "2023-04-24T23:55:18Z")

</div>

Agree with you @CameronBieganek that there is still value in considering the vector-based methods. We are really brainstorming here to see what is the best compromise, nothing decided yet.

> [@CameronBieganek](#):
>
> All that being said, my feeling is that these metric functions are so trivial that there’s not much to be gained by attempting to get Flux and MLJ to use the same loss function implementation. (Is that the implicit goal of LossFunctions.jl?)

It would be extremely helpful for the community to maintain a common set of loss functions in a single repository that is shared among different ML frameworks. It is ideal scenario of course, but I think we should put energy into that direction.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 24, 2023, 11:59pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/26 "2023-04-24T23:59:28Z")

</div>

> [@CameronBieganek](#):
>
> `mean(L1DistLoss(), ŷ, y))`

Regarding aggregation: I only compared this to `value(L1DistLoss(), ŷ, y, AggMode.Mean())`. Sure, you can pack this into `MAE()`, either way.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 25, 2023, 12:10am UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/27 "2023-04-25T00:10:55Z")

</div>

Let me summarize the status quo so that we can organize the discussion moving forward…

Currently we have loss functions in LossFunctions.jl that are represented with structs, and this is useful to hold state and hyperparameters that sometimes exist. Given a `loss` object we can do 3 things with it:

- value: the actual value loss(yhat, y)
- deriv: the first derivative
- deriv2: the second derivative

These derivatives are written by hand, and sometimes are not defined everywhere (autodiff doesn’t help). So LossFunctions.jl has this great feature of collecting non-trivial derivatives besides the value of the function.

Now the design questions regarding iterables of observations `yhat` and `y`, weights `w` and `FUN in [value,deriv, deriv2]`:

1. Should we support `FUN(loss, yhat, y, [w])` and default to `mean` aggregation? Is there any way to inform other types of aggregation in this syntax without dispatching on AggMode types?
2. Should we adopt the alternative syntax `mean(loss, yhat, y, [w])` instead? What about the `deriv` and `deriv2` cases?

It is not clear to me yet which path is the most productive.

Regarding the case where `loss(yhat, y, [w])` returns a vector of values, I agree with @aplavin that we should use broadcast instead. Whichever syntax we choose, we should avoid implicit broadcasting and let Julia handle the “vectorization” for us.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 25, 2023, 12:42am UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/28 "2023-04-25T00:42:21Z")

</div>

The PR is shaping up nicely. I fixed the functor interface for scalar arguments. Now we just need to brainstorm the aggregation API a bit further. It seems to me that the necessity to model deriv and deriv2 will end up forcing us to go with AggMode types, but I may be wrong.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [April 25, 2023, 1:01am UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/29 "2023-04-25T01:01:49Z")

</div>

> [@juliohm](#):
>
> deriv: the first derivative

I’m not sure I understand the utility of `deriv`. In the context of Flux, wouldn’t that be handled automatically by Zygote (or other AD)? In the context of `MLJ.evaluate!`, I don’t think we need `deriv`.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 25, 2023, 1:58am UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/30 "2023-04-25T01:58:03Z")

</div>

Some loss functions are defined by parts and are not dofferentiable everywhere. LossFunctions.jl writes the derivatives manually but people are free to use autodiff as well on top of the value(loss, yhat, y) when that is possible.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [April 25, 2023, 8:55am UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/31 "2023-04-25T08:55:51Z")

</div>

AFAIK, the `ChainRules` package is the go-to way to define custom derivatives. It’s supported by lots of packages, and utilized by multiple AD systems.  
Why not define LossFunctions derivatives through ChainRules? Then there’s no need for separate `deriv` – neither for developers, nor for users to learn.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 25, 2023, 1:33pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/32 "2023-04-25T13:33:52Z")

</div>

I will take a look at ChainRules.jl. Maybe the best plan forward is to consider the API for the value function with sum and mean like in StatsBase.jl and leave the refactoring of deriv and deriv2 for a second PR with ChainRules.jl

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 25, 2023, 6:49pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/33 "2023-04-25T18:49:08Z")

</div>

I’ve updated the PR to adopt the simple sum and mean functions as opposed to the AggMode types. All tests are passing. We just need to update the docs and add tests for iterables that are not AbstractVector.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 27, 2023, 7:10pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/34 "2023-04-27T19:10:32Z")

</div>

The PR has been merged now with the AggMode module gone 🙏

Next I would like to remove the `value` function in favor of the functor interface always.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 27, 2023, 9:20pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/35 "2023-04-27T21:20:24Z")

</div>

Done. If anyone wants to take a look at the master branch of LossFunctions.jl I think we are ready for another breaking release with removed AggMode and removed value function. Anything else we could break right away before the next release?

The work around deriv and deriv2 I will leave for a future opportunity. I am not used to the ChainRules.jl ecosystem, and someone can help there.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [April 28, 2023, 4:43pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/36 "2023-04-28T16:43:14Z")

</div>

@aplavin do you have an example of how we could replace deriv and deriv2 by the ChainRules.jl approach? I could then copy/paste the changes throughout the package.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [May 3, 2023, 5:21pm UTC](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738/37 "2023-05-03T17:21:55Z")

</div>

I’ve released LossFunctions.jl v0.10 with the latest breaking changes.

[Previous page](https://discourse.julialang.org/t/request-to-upgrade-to-lossfunctions-jl/97738.md?page=1)