Late evaluation of expression as anymous function

sstroemer · February 12, 2023, 3:06pm

I want to iteratively build a function, but I can’t figure out how. I’ll try and give a non-working example hoping that this helps explain what I am trying to do.

mydata = Dict(:a => 1, :b => 2, :c => 3)

function build_f(myflag::Bool)
    # start of the expression as `1.0 + _input[:a]`
    ex = Expr(:call, :+, 1.0, :(_input[:a]))
    
    if myflag
        # add `2.0 * _input[:b]` to the expression
        ex = Expr(:call, :+, ex, Expr(:call, :*, 2.0, :(_input[:b])))
    else
        # add `2.0 * _input[:c]` to the expression
        ex = Expr(:call, :+, ex, Expr(:call, :*, 2.0, :(_input[:c])))
    end

    # return a function mapping some input dictionary to
    # the result of the expression
    (_input) -> eval(ex)
end

# build the function, which evaluates:
# `1.0 + _input[:a] + 2.0 * _input[:b]`
f = build_f(true)

f(d) # fails with `UndefVarError: _input not defined`

What I am trying to get is something that I can use like:

f(d)       # = 1 + 1 + 2*2 = 6
d[:b] = 3  # update the dict
f(d)       # now = 1 + 1 + 2*3 = 8

jling · February 12, 2023, 3:15pm

one option is to use GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel execution

you can build them and only fetch() the returned object which will compute only needed nodes

sstroemer · February 12, 2023, 3:20pm

Can you point me into a specific direction with Dagger.jl? I’m lost on how that would help me build up functions and evaluate them (I am not looking to distribute anything).

jling · February 12, 2023, 3:33pm

if all you need is changing value, you can use closure:

julia> d = Dict(:a => 1, :b => 2, :c => 3);

julia> function build()
           () -> 1 + 1 + d[:b]
       end
build (generic function with 1 method)

julia> f = build();

julia> f()
4

julia> d[:b] = 3
3

julia> f()
5

this comes with performance penalty be warned.

what are you ACTUALLY trying to do? this sounds like a XY problem

bertschi · February 12, 2023, 3:35pm

Most of the time eval is probably not the right idea (and often does not work as intended since it evaluates in the global scope only).
Would regular currying or partial application work in your case?

function builder(myflag::Bool)
    common(_input) = 1.0 + _input[:a]
    if myflag
        _input -> 2.0 * _input[:b] + common(_input)
    else
        _input -> 2.0 * _input[:c] + common(_input)
    end
end

julia> d = Dict(:a => 1, :b => 2, :c => 3)
Dict{Symbol, Int64} with 3 entries:
  :a => 1
  :b => 2
  :c => 3

julia> f = builder(true)
#1 (generic function with 1 method)

julia> f(d)
6.0

julia> d[:b] = 3
3

julia> f(d)
8.0

sstroemer · February 12, 2023, 3:58pm

That looks a lot like what I was trying to achieve, but comes at one problem: the first call to f(d) has a considerable overhead due to compilation, right? And if I am creating many of those functions and only evaluating each a couple of times, the compile time will be bad.

~~Is there a way to circumvent this? (that’s the reason I thought about going for expressions)~~

It shows… My minimal (non) working example did not really specify what I need. The “builder” actually needs to iteratively construct the returned function/expression/… I am sorry for not clarifying that. Because that does not work with the current approach, see:

function builder(myflag::Bool)
    common(_input) = 1.0 + _input[:a]

    if myflag
        common = _input -> 2.0 * _input[:b] + common(_input)
    else
        common = _input -> 2.0 * _input[:c] + common(_input)
    end

    if myflag
        common = _input -> 5.0 * _input[:c] + common(_input)
    else
        common = _input -> 5.0 * _input[:b] + common(_input)
    end

    common
end

f = builder(true)
f(Dict(:a => 1, :b => 2, :c => 3))

which results in a ERROR: StackOverflowError: (due to the circular reference).

bertschi · February 12, 2023, 5:50pm

Don’t quite get why it has to be done iteratively, but there are several options:

You could use let to introduce a new scope and thereby nest the constructed closures:

function builder(myflag::Bool)
    let common = _input -> 1.0 + _input[:a]
    
    let common = if myflag
             _input -> 2.0 * _input[:b] + common(_input)
        else
             _input -> 2.0 * _input[:c] + common(_input)
        end

    let common = if myflag
             _input -> 5.0 * _input[:c] + common(_input)
        else
             _input -> 5.0 * _input[:b] + common(_input)
        end

    common
end
end
end
end

Compose small builder functions to extend the constructed function, a la fluid interfaces:

julia> step1() = _input -> 1.0 + _input[:a]
step1 (generic function with 1 method)

julia> step2(myflag::Bool) = common -> if myflag; _input  -> 2.0 * _input[:b] + common(_input) else _input -> 2.0 * _input[:c] + common(_input) end
step2 (generic function with 1 method)

julia> step3(myflag::Bool) = common -> if myflag; _input  -> 5.0 * _input[:c] + common(_input) else _input -> 5.0 * _input[:b] + common(_input) end
step3 (generic function with 1 method)

julia> f = step1() |> step2(true) |> step3(true)
#57 (generic function with 1 method)

julia> f(Dict(:a => 1, :b => 2, :c => 3))
21.0

Use eval on an expression that returns your desired function:

function build_f(myflag::Bool)
    # start of the expression as `1.0 + _input[:a]`
    ex = Expr(:call, :+, 1.0, :(_input[:a]))
           
    if myflag
        # add `2.0 * _input[:b]` to the expression
        ex = Expr(:call, :+, ex, Expr(:call, :*, 2.0, :(_input[:b])))
    else
        # add `2.0 * _input[:c]` to the expression
        ex = Expr(:call, :+, ex, Expr(:call, :*, 2.0, :(_input[:c])))
    end

    # return a function mapping some input dictionary to
    # the result of the expression
    eval(:(_input -> $ex))
end

Performance-wise eval is probably not better than any of the other approaches … in any case, the constructed function has to be compiled before being executed and eval will need to do everything at runtime.

stevengj · February 12, 2023, 5:52pm

What is the context here? Why can’t you simply use higher-order functions, which are almost always far superior to working manually with symbolic expressions?

sstroemer · February 12, 2023, 6:36pm

Yes, thanks! That works. Now I need to somehow figure out a more performant approach …

What I am trying to do is taking a (somewhat “user generated”) configuration and iteratively building a set of functions. I do have a large “list” of configurations (that I can only iterate once), and each one decides which “parameter” (my dictionary in the example) is added to which of the final calculation function. This results in n “built up” functions, that I can repeatedly evaluate - based on some input that is changed between iterations.

So I am looking to generate f_i(x) := \alpha_i + \sum_{j \in J_i}\beta_{i,j}\cdot x_j where J_i (some subset of all available entries in x) as well as the constant (\alpha_i) and the coefficients (\beta_{i,j}) vary for each f_i.

After building these functions, I am repeatedly evaluating them for different x (but the functions stay fixed). While it’s mostly about the performance of the evaluation, the number n of functions can easily be > 10^8… Therefore the performance of building these f_i (as well as the memory usage) matters somewhat.

bertschi · February 12, 2023, 7:01pm

In that case, I would probably use always the same function, i.e., f(α, β, x) = α + β * x and construct suitable sparse vectors/matrices for α and β.
You might also want to look if the formula interface of StatsModels.jl might be helpful.

sstroemer · February 12, 2023, 7:21pm

That looks interesting. Thanks for the hint!

uniment · February 12, 2023, 7:35pm

Have you seen the section in the language manual on functors?

You might consider something along these lines:

using LinearAlgebra

struct WeirdFun{Α<:Number, Β<:AbstractVector{<:Number}} <: Function
    α :: Α
    β :: Β
end
WeirdFun(myflags::AbstractVector{Bool}) = ... # constructor---calculate α, β then call WeirdFun(α, β)

(wf::WeirdFun)(x::AbstractVector{<:Number}) = wf.α + dot(wf.β, x)

f = WeirdFun(...)
f(...)