How often is Loss Function executed

New to Julia
1

I am beginner in Julia and try to understand the basics.
I have an ODE problem with 2 parameters (PI control). These are to be learned from a measurement series.

My loss function looks like this:

function loss(p)             
     prediction  = solve(prob_nn, Euler(), dt=Δt, p=p, saveat = t, sensealg=ForwardDiffSensitivity(convert_tspan=true))
     loss = sum(abs2, prediction[1:end-1,:].-X)     
     return loss, prediction
end

and it is called by sciml_train:

result_ode = DiffEqFlux.sciml_train(loss, G_nn, BFGS())                 #, cb = callback)

The algorithm finds the desired parameters in 8 iterations. Now I would like to know how often my loss function is actually called. I suspect that will happen more than 8 times (not just for calculating the gradients).
How can I find out? I wanted to use Timeroutputs, but don’t know how. I can’t place the timer into the loss function like

@timeit to "loss_time" loss = sum(abs2, prediction[1:end-1,:].-X)

nor in sciml_train.
My next approach was the instrumenting profiler, but the IProfile package seems to be no longer available.
I would be very grateful if someone could help me with this problem.

2

One general way I would do it is: define a callable structure, which holds a counter that gets increased any time it gets called.

For example you can wrap your loss in the following:

mutable struct Counting{T}
    f::T
    counter::Integer
end

(c::Counting)(x) = begin
    c.counter += 1
    c.f(x)
end

So instead of passing loss to whomever needs to call it, you pass Counting(loss, 0) and then go check the value of the counter at the end of the computation.

Edit: I interpreted your question as “how many times”, but I realized that by “how often” you mean to measure how much time passes between one call and the next. You can adapt the snippet above to record times using whatever timer mechanism, instead of counting.

3 Likes
3

I don’t have the packages to run this on my machine. But, I think you can try

const to = TimerOutput();

@timeit to "loss_function" function loss(p)             
     prediction  = solve(prob_nn, Euler(), dt=Δt, p=p, saveat = t, sensealg=ForwardDiffSensitivity(convert_tspan=true))
     loss = sum(abs2, prediction[1:end-1,:].-X)     
     return loss, prediction
end

result_ode = DiffEqFlux.sciml_train(loss, G_nn, BFGS())   

show(to)
4

another (ugly) way for counting the calls:

function mkfun()
  noc=0
  getnoc()=noc
  function fun(a,b)
    noc+=1
    a+b
  end
  fun,getnoc
end


f,fnoc=mkfun()
g,gnoc=mkfun()
h(a,b)=0
funarr=[f,g,h,f]
for k in 1:1000
  funarr[rand(1:length(funarr))](rand(1:10),rand(1:10))
end
println("f: ",fnoc()) # about a half
println("g: ",gnoc()) # about a quarter

drawback: you have to include the noc+=1 into your function. :frowning:

If I were you, I would follow @lostella’s more general solution, which is - in my understanding - could be the following:

function myfun0()
  sleep(0.1*rand())
end

function myfun1(x)
  sleep(0.2*x)
end

function myfun2(x,y)
  sleep(x*y)
end

struct Cnt{F}
  f::F
  callsat::Array{Float64}
end

function (c::Cnt)(x...)
  push!(c.callsat,time())
  c.f(x...)
end


function caller(fun0,fun1,fun2,n)
  for k in 1:n
    r=rand()
    if r<1.0/3.0
      fun0()
    elseif r<2.0/3.0
      fun1(rand())
    else
      fun2(rand(),rand())
    end
  end
end  

cmyfun0=Cnt(myfun0,Float64[])
cmyfun1=Cnt(myfun1,Float64[])
cmyfun2=Cnt(myfun2,Float64[])

caller(cmyfun0,cmyfun1,cmyfun2,10)

[cmyfun0.callsat, cmyfun1.callsat, cmyfun2.callsat] .|> length |> println

Actually I never used structs or callable structs before, but they are very usable - worth to learn them!

1 Like
5

A functional style solution for creating a counter:

function counter(f)
    count = Ref(0)
    function (args...; kwargs...)
        count[] += 1
        f(args...; kwargs...)
    end, count
end

counting_loss, cnt = counter(loss)

But if this is just a one-time experiment to see what’s happening, it might be fine to simply increment a global

counter = 0
function loss(p)
  global counter += 1
  ...
end
2 Likes
6

Thanks for this idea. This is exactly what I was looking for. However, I did something wrong while trying it out and can’t find my mistake.
I tried

struct Counting{T}
    f::T
    counter::Integer
end

(c::Counting)(x) = begin
    c.counter += 1
    c.f(x)
end

function testf(y)
y^2
end

cntr = Counting(testf, 0)

cntr(1.2)

But I can’t change the value of the counter. I get:
ERROR: LoadError: setfield! immutable struct of type Counting cannot be changed

7

You want mutable struct

8

Hello Kristoffer,
but why does czylabsonasa’s version (2. suggestion) work? It’s not a problem to concatenate to ‘callsat’.

9

yeah I fixed my snippet to add mutable, sorry about that, I didn’t really run it back then

10

Ah, then one can change an array in an immutable struct because the array itself is mutable (just a reference), but not a value? I think that I have even read that.

11

I think the answer is: if your intention is to reassign of some member of your struct then you should use mutable (for the primitive type Int the expression cntr+=1 is reassignment, we need mutable) but there is no reassignment for callsat in the above example, you only mutate its content. But maybe some expert will explain you better.
read this: Assignment and mutation - #4 by StefanKarpinski
and this: Julia Learning Circle: Memory Allocations and Garbage Collection · W.

1 Like
12

Thank you for this approach. I don’t want to use a global variable. But the first solution fits my purposes. Unfortunately, I can only mark one answer as a solution.