# Extract intermediate data from inside a function

**URL:** <https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251>\
**Category:** General Usage\
**Created:** [December 3, 2018, 2:58pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251 "2018-12-03T14:58:32Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Philippe\_Maincon1](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@Philippe\_Maincon1](https://discourse.julialang.org/u/Philippe_Maincon1)\
**Post date:** [December 3, 2018, 2:58pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/1 "2018-12-03T14:58:32Z")

</div>

Hi,  
here comes a question which in any other language would be wishful, yet in Julia, I dare wish.

So I program a function  
R = f(X)  
think finite elements 101, given nodal displacements, the function returns nodal forces.

Of course, there are many intermediate results, say strain, and stress, not needed outside of my function f, in the logic of the FEA solver, but valuable for debugging and not least, for reporting.

My strategy is to _not_ store intermediate results systematically during the analysis. This gobbles memory and CPU, and fills the code with boilerplate. What I want to do is store all the X (for all elements and steps…), and if I want, call f after the analysis is complete. And now the crux: while adding as little boilerplate as possible, I want to be able to “call f(X)” and add “give me the stresses”.

How would I do that?

- store intermediate results in a struct, return the struct, and the solver just throws it away
- metaprogramming? Transform _readable_ code for function f into code with boilerplate for the reporting of intermediate results?
- …???

With what strategy would you attack the problem? I am not asking for code, but pointers to the relevant language features that could be exploited here.

: )

Philippe

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 3, 2018, 3:15pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/2 "2018-12-03T15:15:28Z")

</div>

Similar to the `eigs` function, you could return a tuple of results, with the most commonly needed results first, in the expectation that most callers will ignore the subsequent results. For example, `x, = f(...)` discards all results except for the first.

Or you could pass some optional keyword argument, with default value nothing—if the caller passes an empty array (or some other data structure with mutable contents) instead, it is mutated by your function to hold the requested data.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 3, 2018, 3:25pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/3 "2018-12-03T15:25:07Z")

</div>

Use a thunk?

```julia
function f(x)
    y = 2*x
    calculate_extra = () -> 3*y
    y, calculate_extra
end

# just the result
y, = f(1)

# extras calculated on demand
y, g = f(1)
g()

```

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2018, 3:56pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/4 "2018-12-03T15:56:26Z")

</div>

> [@Tamas\_Papp](#):
>
> # extras calculated on demand

looks like the calculation happens regardless, so it would be more accurate to say **assigned** instead of _calculated_ on demand, since the calculations are not affected

```Julia
julia> function f(x)
           y = 2*x
           calculate_extra = (println("calculated some extra"); () -> 3*y)
           y, calculate_extra
       end
f (generic function with 1 method)

julia> y, = f(1)
calculated some extra
(2, getfield(Main, Symbol("##9#10")){Int64}(2))

```

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2018, 4:12pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/5 "2018-12-03T16:12:43Z")

</div>

just write it as

```julia
           calculate_extra = () -> (println("calculated some extra"); 3*y)

```

The calculations are indeed affected.

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2018, 4:13pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/6 "2018-12-03T16:13:59Z")

</div>

> [@kristoffer.carlsson](#):
>
> The calculations are indeed affected.

Depends on what you are trying to do, my point is that `calculate_some` extra is created either way, since it is an anonymous function you can defer some calculation for later if you want to, but this function is created regardless also if you don’t use it. I consider the creation of it as a calculation.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [December 3, 2018, 4:15pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/7 "2018-12-03T16:15:12Z")

</div>

> [@chakravala](#):
>
> Depends on what you are trying to do

Do extra calculations on demand?

---

<div class="post-metadata">

**Author:** ![bennedich](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bennedich/32/4894_2.png) [@bennedich](https://discourse.julialang.org/u/bennedich)\
**Post date:** [December 3, 2018, 4:15pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/8 "2018-12-03T16:15:24Z")

</div>

I’ve had a similar problem, and solved it by having the function return a tuple consisting of the solution and a struct with all intermediates. I then had a `Bool` flag passed to the function, and only calculated/populated the intermediates if the flag was true. (Otherwise, I left those fields empty.)

I’m not quite sure I follow the suggestion of calling `f(X)` a second time to get the intermediates, how would that avoid boilerplate code? Are you imagining a separate code path to calculate those?

Boilerplate code that makes the code less readable is of course an issue. A bigger issue, IMO, is repeated code. In a situation like this, I’d first try to reduce boilerplate code by putting the code that calculates the intermediates in a separate function, then put something like this somewhere in the main algorithm:

```julia
calculate_intermediates && calculate_intermediates!(X)

```

Hard to give more specific advice without seeing your code, perhaps your algorithm looks very different in the case that the intermediates are calculated?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [December 3, 2018, 5:35pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/9 "2018-12-03T17:35:43Z")

</div>

Another possibility is to have a callback function that you call on the intermediate data, which can then decide whether to save it or do something else. For example, consider the following function that iteratively computes a recurrence relation (e.g. a discretized ODE or PDE):

```julia
function iterate_recurrence!(usercallback::Function, x, N)
    usercallback(x, 0) # initial data
    for n = 1:N
        some_recurrence!(x, n) # mutate x somehow, in-place
        usercallback(x, n)
    end
    return x # return final result only
end

# define a default callback that does nothing
iterate_recurrence!(x, N) = iterate_recurrence!((x,n)->nothing, x, N)

```

The default `usercallback` does nothing, and only the final result of the recurrence is returned. You certainly wouldn’t want to store all of the intermediate `x` values by default just in case the user might want them, because this could incur a huge storage cost (`x` could be a huge vector and `N` could be huge).

The advantage of a functional style (a `usercallback` function) is that then the caller has complete freedom to decide what to do. For example, you can use a callback to save every 10th `x`:

```julia
function doit(x, N)
    xsaved = [copy(x)] # array to store saved x's, initialized with first x
    iterate_recurrence!(x, N) do xₙ, n
        if n % 10 == 0 && n > 0
            push!(xsaved, copy(xₙ))
        end
    end
    return x, xsaved
end

```

Note that putting the callback as the _first_ argument allowed me to use [Julia’s nice `do` syntax](https://docs.julialang.org/en/v1/manual/functions/#Do-Block-Syntax-for-Function-Arguments-1). Also note that I pushed `copy(xₙ)`, _not_ `xₙ`, to the array `xsaved` because `iterate_recurrence!` mutates `x` in-place—I want to save `N÷10` distinct vectors, not `N÷10` references to the same vector.

You could imagine lots of other variations. For example, the callback could save only one _component_ `x[1]` of the data. Or it could update a plot of the data. Or it could write data to the disk. It would not be possible to provide all possible behaviors efficiently just by passing flags to `iterate_recurrence!`, but a callback interface offers this flexibility with ease.

A much more sophisticated variation on this idea is implemented by the DifferentialEquations.jl package, which provides a [wide variety of event-driven callback interfaces](http://docs.juliadiffeq.org/latest/features/callback_functions.html).

---

<div class="post-metadata">

**Author:** ![chakravala](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chakravala/32/6832_2.png) [@chakravala](https://discourse.julialang.org/u/chakravala)\
**Post date:** [December 3, 2018, 6:30pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/10 "2018-12-03T18:30:12Z")

</div>

One technique which I employed in [SyntaxTree.jl](https://github.com/chakravala/SyntaxTree.jl) is to have intermediate helper functions which get called from a meta-function for organizing the shared allocations.

[https://github.com/chakravala/SyntaxTree.jl/blob/master/src/exprval.jl](https://github.com/chakravala/SyntaxTree.jl/blob/master/src/exprval.jl)

In this case, the `expravg` function computes some intermediate values needed in several later calculations:

```nohighlight
function exprval(expr)
    val = expravg(expr)
    cal = callcount(expr)
    mal = sqrt(exprdev(expr,val[2],cal))
    cal*sqrt(abs(val[2])*mal)*val[4], cal, mal, val[2], val[4]
end

```

Then all the different values from the helper functions are combined in the meta-function. However, I only export the final meta-function and not the helper functions.

What’s special about this is that the helper function called `expravg` is both capable of recursively allocating itself to tally up calculations from an abstract syntactic tree and it can also be used for obtaining the intermediate calculations in the meta-function, thus splitting off that functionality neatly into a separate recursive function which gets called from and terminates at the meta-function needing intermediate result.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [December 3, 2018, 6:34pm UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/11 "2018-12-03T18:34:14Z")

</div>

I might have thought that the multiplication would be optimized away. I don’t see how to change the value of `y` after the call returns.

So `f(1)` would return

```julia
2, () -> (println("calculated some extra"); 6)

```

But, that’s not the case

```julia
julia> function f(x)
                  y = 2*x
                  calculate_extra = (println("calculated some extra"); () -> 3*y)
                  y, calculate_extra
              end
f (generic function with 1 method)

julia> z, g = f(1)
calculated some extra
(2, getfield(Main, Symbol("##5#6")){Int64}(2))

julia> @code_lowered g()
CodeInfo(
1 ─ %1 = (Core.getfield)(#self#, :y)
│ %2 = 3 * %1
└── return %2
)

```

---

<div class="post-metadata">

**Author:** ![Philippe\_Maincon1](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@Philippe\_Maincon1](https://discourse.julialang.org/u/Philippe_Maincon1)\
**Post date:** [December 4, 2018, 7:33am UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/12 "2018-12-04T07:33:24Z")

</div>

Hi Bennedich,

To explain my multiple pass idea: in the analysis, “f” is called for each element at each iteration of each step.

After the analysis, I want to allow the user to extract arbitrary “intermediaries”, interactively. But I want to avoid to store all intermediaries during the analysis, hence the extra calls to “f” “directly” by the user. So extra calls is not a strategy to solve the given problem, I make it part of my problem definition.

: )

Philippe

---

<div class="post-metadata">

**Author:** ![Philippe\_Maincon1](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@Philippe\_Maincon1](https://discourse.julialang.org/u/Philippe_Maincon1)\
**Post date:** [December 4, 2018, 7:49am UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/13 "2018-12-04T07:49:19Z")

</div>

Neat.

Very neat. With, in addition, a little macro to hide the calls to the callback, a line in “f” could look like

> function f(usercallback::Function,X, morestuff)  
> @remember intermediate\_result = some\_hack(X)  
> I will think about this very carefully, thank you!

: )

Philippe

---

<div class="post-metadata">

**Author:** ![Philippe\_Maincon1](https://avatars.discourse-cdn.com/v4/letter/p/ec9cab/32.png) [@Philippe\_Maincon1](https://discourse.julialang.org/u/Philippe_Maincon1)\
**Post date:** [December 4, 2018, 8:00am UTC](https://discourse.julialang.org/t/extract-intermediate-data-from-inside-a-function/18251/14 "2018-12-04T08:00:15Z")

</div>

…and a reply to all: I was right to ask, I got valuable input! In several cases, your answer points at features of Julia I am not well familiar with, so it will take me some time to understand in depth how you think.

But I am sure, that by the time I have done that, I will see one or more good solution to my question.

Thank you indeed!