# Change of return syntax for loss\_function in Optimization.jl

**URL:** <https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224>\
**Category:** Optimization (Mathematical)\
**Tags:** question, optimization\
**Created:** [November 28, 2024, 4:50pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224 "2024-11-28T16:50:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![BambOoxX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bambooxx/32/22179_2.png) [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)\
**Post date:** [November 28, 2024, 4:50pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/1 "2024-11-28T16:50:20Z")

</div>

I’m using Optimization.jl to optimize conditions of ODE simulation to a given objective.  
Before the latest update (4.0.0), one could use this return syntax

```julia
function loss_adjoint(fullp, batch, time_batch)
    pred = predict_adjoint(fullp, time_batch)
    sum(abs2, batch .- pred), pred
end

```

allowing to retrieve some object, in this case `pred` at each solver iteration.  
In the latest version, it seems this syntax is no longer available

```julia
function loss_adjoint(fullp, data)
    batch, time_batch = data
    pred = predict_adjoint(fullp, time_batch)
    sum(abs2, batch .- pred)
end

```

What would be the proper way to retrieve `pred` with the newest version ?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 28, 2024, 5:20pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/2 "2024-11-28T17:20:23Z")

</div>

> [@BambOoxX](#):
>
> What would be the proper way to retrieve `pred` with the newest version ?

Just enclose it or use a global.

---

<div class="post-metadata">

**Author:** ![BambOoxX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bambooxx/32/22179_2.png) [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)\
**Post date:** [November 28, 2024, 5:31pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/3 "2024-11-28T17:31:23Z")

</div>

What do you mean by enclose it ?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 28, 2024, 5:55pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/4 "2024-11-28T17:55:18Z")

</div>

Capture it via a closure or use a callable struct.

---

<div class="post-metadata">

**Author:** ![BambOoxX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bambooxx/32/22179_2.png) [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)\
**Post date:** [November 28, 2024, 6:13pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/5 "2024-11-28T18:13:12Z")

</div>

I am sorry I do not see what closure to implement in this case. Could you point me to a relevant example in the docs. In that sense, the new [Data Iterators and Minibatching · Optimization.jl](https://docs.sciml.ai/Optimization/stable/tutorials/minibatch/) example is less informative than the former one.

---

<div class="post-metadata">

**Author:** ![BambOoxX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bambooxx/32/22179_2.png) [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)\
**Post date:** [November 29, 2024, 1:28pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/6 "2024-11-29T13:28:55Z")

</div>

Ok, so I read [Common Solver Options (Solve Keyword Arguments) · Optimization.jl](https://docs.sciml.ai/Optimization/dev/API/solve/#CommonSolve.solve-Tuple%7BOptimizationProblem,%20Any%7D) and in that case, an additional `predict` is used twice to retrieve `pred` in both the loss evaluation and in the callback, using the current `OptimizationState`.

```julia
function predict(u)
    Array(solve(prob, Tsit5(), p = u))
end

function loss(u, p)
    pred = predict(u)
    sum(abs2, batch .- pred), pred
end

callback = function (state, l; doplot = false) #callback function to observe training
    display(l)
    # plot current prediction against data
    if doplot
        pred = predict(state.u)
        pl = scatter(t, ode_data[1, :], label = "data")
        scatter!(pl, t, pred[1, :], label = "prediction")
        display(plot(pl))
    end
    return false
end

```

This works, but is this what you had in mind ? Calling `predict` twice seems like a bad design.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 29, 2024, 4:58pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/7 "2024-11-29T16:58:33Z")

</div>

```julia
pred = Ref{Any}()

function predict(u)
    Array(solve(prob, Tsit5(), p = u))
end

function loss(u, p)
    pred[] = predict(u)
    sum(abs2, batch .- pred[])
end

callback = function (state, l; doplot = false) #callback function to observe training
    display(l)
    # plot current prediction against data
    if doplot
        pl = scatter(t, ode_data[1, :], label = "data")
        scatter!(pl, t, pred[][1, :], label = "prediction")
        display(plot(pl))
    end
    return false
end

```

Etc. you can optimize different ways of doing it from there.

---

<div class="post-metadata">

**Author:** ![BambOoxX](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bambooxx/32/22179_2.png) [@BambOoxX](https://discourse.julialang.org/u/BambOoxX)\
**Post date:** [November 29, 2024, 5:32pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/8 "2024-11-29T17:32:15Z")

</div>

Thanks @ChrisRackauckas ! Just a side question for which I have no use right now, but for the sake of the discussion, I’m guessing this approach is not compatible with [Multistart optimization with EnsembleProblem · Optimization.jl](https://docs.sciml.ai/Optimization/stable/tutorials/ensemble/) or is it ?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 29, 2024, 7:48pm UTC](https://discourse.julialang.org/t/change-of-return-syntax-for-loss-function-in-optimization-jl/123224/9 "2024-11-29T19:48:19Z")

</div>

You can definitely do this kind of thing with multistart, but if you batch multithread then you need to take that into account in the callbacks