Metaprogramming: substitute the name of the argument

New to Julia
,
1

I am writing a function factory that takes two functions and weighs them by some _wt \in (0,1).

function fun_mix(fun1::Function, fun2::Function; _nm_wt="_wt")
       f = function (u::AbstractFloat, args...; _wt=0.5)
          _wt * fun1(u; args...) + (1 -_wt) * fun2(u; args...)
        end
     return f
end

Now, I want to allow the user to change the default value of _nm_wt to be something else, say beta, so that later the inner function could be called with that new parameter name and not the placeholder name I proposed here.

Essentially I want to capture function f parse the AST, find all the places where _wt is used and substitute it with the Symbol(_nm_wt). Note, that if I capture the function definition at the time of creation, then I have to capture and substitute the function names fun1 and fun2 and, potentially, all the things they point to, so I would prefer to not mess with that, if possible.

In R, you can capture a function and manipulate its formals and body separately, so I have done it as

fun_mix <- function(fun1, fun2, nm_wt=".wt"){
   f <- function(u, .wt=0.5, ...){
    (.wt)*fun1(u, ...) + (1-.wt)*fun2(u, ...)
   }

  formals_ <- formals(f)
  body_ <- body(f)
  names(formals_)[names(formals_) == ".wt"] <- nm_wt
  body_ <- do.call(substitute, list(body_, list(.wt = as.symbol(nm_wt))))
  as.function(c(formals_, body_))
}

How do I repeat this success in Julia?

2

You cannot do this in Julia because Julia functions don’t carry their AST’s around with them, so you can’t ask for its body.

Stepping back – forget that R has a tradition of doing this sort of thing. What is your use case in terms of user-facing behaviors? If, in your example, you want to change _wt to a custom name, you can splice in _nm_wt to the body of f once you rewrite fun_mix as a macro. Then the end user will get back a function with the keyword argument of their choice.

Stepping back further, the example you’ve given is not a macro, so you can’t do syntactic rewrites in it. Have you read an intro to metaprogramming in Julia? That may be useful background before trying to do cross-language comparisons.

3 Likes
3

Yes, @johnmyleswhite. Sorry about the novice question! I skimmed through the introduction, did not see anything on capturing functions and therefore decided to ask.

Writing macros is probably inevitable here, but I tried to see it I could stay within the familiar territory of combining functions. I seem to find some traces of function manipulation here.

The use-case is that I want to have a collection of function factories that manipulate function calls to produce more complex functions from simpler functions (like this one re-weights two functions). The use would write something like this (in meta-code):

big_function = small_fun1 |>
    @transformation |>
    @anothertransformation |>
    @fun_mix(small_fun2, nm_wt="beta")

big_function(u, beta=0.1)

where small_fun1(u, args...) and small_fun1(u, args...) are some functions of the same input u

I am learning and thanks for your patience!

4

Here is a macro that should likely does NOT (see @Benny below) work, if I understood you correctly:

macro funmix(f, g, nm_wt=:_wt)
	nm_symbol = esc(nm_wt)
	quote
		# evaluate the function definitions only once!
		let f = $f, g = $g
			(u, args...; $(nm_symbol)=0.5) -> $(nm_symbol)*f(u, args...) + (1-$(nm_symbol))*g(u,args...)
		end
	end
end

Use it like:

julia> fmix = @funmix(sin, cos, asdf);
julia> fmix(Ď€; asdf=0.5)
-0.5

# alternative
julia> fmix2 = @funmix(sin, cos);
julia> fmix(Ď€; _wt=0.5) # default name

BTW: This

cannot work because macros are expanded before function calls are applied. So you cannot use |> (a function) to pipe some argument into a macro.
You’ll need to use @chain from Chain.jl or @pipe from Pipe.jl for this.

2 Likes
5

You don’t generally need metaprogramming for this. Just use higher-order functions.

For example, this seems to match the pattern you want?

fun_mix(fun1, fun2) = (u; wt=0.5, kws...) -> wt * fun1(u; kws...) + (1-wt) * fun2(u; kws...)

giving e.g.

julia> fmix = fun_mix(sin, cos)
#19 (generic function with 1 method)

julia> fmix(0.3)
0.6254283478934728

julia> fmix(0.3; wt=0.9)
0.3615018349077662
6 Likes
6

Long story short, don’t.

As you’ve acknowledged, you’re mixing up some things here. First let’s clear up what macros do. Source code is parsed until reaching a complete top-level expression, any macro calls transform the expression (still considered parse-time), each expression is evaluated into the program, and repeat. It is fundamentally impossible for macro calls in the body of fun_mix to access the runtime value of _nm_wt when fun_mix is called because the macros do their job during the parsing phase of the function fun_mix ... end definition.

Second, let’s talk about how closures (inner function) are implemented compared to top-level functions. A function has its own concrete type, a subtype of the abstract type Function, and typical top-level ones like fun_mix get singleton types, in other words the function is the only instance of said type. But closures usually aren’t singleton instances because different instances would capture different values. The thing is, the different instances still share the same type; in fact, when fun_mix was defined and got its type, the closure was also defined and got a type. In both cases, the function types are in the global scope associated with the methods you defined. (Incidentally, this is why conditional methods don’t work for closures and aren’t intended to, but the perk is you only need to compile per method, not per closure instance). When you call fun_mix, you’re just instantiating the closure’s type, and immutable instances capturing the same values would be the same instance; it’s entirely different from R where each call defines a separate closure every call, so closures that capture the same values would still be different (not identical).

Like johnmyleswhite already said, methods don’t carry around the expressions, so you can’t go in and edit it, only replace it. But even if you could, there’s only one method so the change is present everywhere; you don’t have one with a keyword of _wt and another with a keyword of beta simultaneously. To emulate R closures, you have to do runtime metaprogramming: start with an Expr, insert the functions (evaluated instances work best, the temporary names fun1 and fun2 won’t work at all) and keyword, and evaluate it (eval must evaluate code into the global scope no matter where it’s called), making a separate function each time. The minor issue is now you have to compile almost the exact same code for each function. The major issue is all function’s types are currently assigned to generated const global variables and thus persist for the entire Julia process even if the function instances are long gone, so you don’t want to do this arbitrarily many times.

Zooming out a bit, you might think Julia is a bit less dynamic than R and other common dynamically typed languages like Python, and you’d be right. Julia’s still plenty dynamic for interactive workflows, but there’s several sacrifices to allow optimizing compilation to work. Go for consistent and documented keyword usage within a multimethod function and possibly among a set of related functions instead of letting the user build their own API. Piecing together simpler functions should be documented in code examples or arranged into a higher-order function.

2 Likes
7

Thanks a lot! This is very illuminating. I could get by with higher-order functions and well documented arguments, of course. I am just hedging against the argument name collisions when the same transformation is applied more than once.

Perhaps my weird package will have to stay in R for now. Thanks for everyone’s input!

8

Why is this a problem with higher-order functions? For example, with the fun_mix example, one can easily apply the same transformation twice:

julia> fmix = fun_mix(sin, cos);

julia> fmix2 = fun_mix(abs, u -> fmix(u; wt=0.9));

julia> fmix2(1.2)
1.0375354764090856

There is no collision because the argument names are scoped.

9

That would be nice @stevengj , but I want to determine the values for all named arguments at the time when I call fmix2, such as

fmix2(1.2, wt1=0.3, wt2=0.1)
fmix2(1.1, wt1=0.6, wt2=0.001)
...

and have them “land” in appropriate slots in the chain of function transformations. Thank you for suggestion, though!

10 
fmix2 = (u; wt1=0.5, wt2=0.9) -> fun_mix(abs, u -> fmix(u; wt=wt2))(u; wt=wt1)

gives

julia> fmix2(1.2, wt1=0.3, wt2=0.1)
0.6535281213380102
2 Likes
11

Wow, neat thank you! Now I need to think how to wrap it into a user-friendly API. Things get hairy pretty quickly after 3-4 transformations.

Thank you!

12

I would suggest you try other people’s higher-level suggestions, first. But if you really insist on this functionality, you can still do it without metaprogramming:

julia> function fun_mix(fun1::Function, fun2::Function; _nm_wt=:_wt)
             f = function (u::AbstractFloat, args...; kwargs...)
                _wt = kwargs[_nm_wt]
                _wt * fun1(u; args...) + (1 -_wt) * fun2(u; args...)
             end
          return f
       end
fun_mix (generic function with 1 method)

julia> fun_mix(sin, cos; _nm_wt=:beta)(0.3; beta=1//3)
0.7353977283041838

Note that here I have used a :beta (a Symbol) rather than "beta" (a String), since that is how keyword arguments are labeled. What I’ve done works because vararg keyword arguments get passed as an indexable container, so I simply look up the requested key in that container.

This method has a fair amount of runtime overhead, so I don’t really recommend it where performance matters.

1 Like
13

Reading the linked package, it’s a heavily metaprogramming approach to manipulating quantile functions (in the mathematical sense, so I’ll just abbreviate to QFs and say “function” in the sense of the respective language from now on). You start with precursor functions*, you modify and mix them with functions representing the QF transformation rules, then manually add custom-named parameters to make working functions to compute QF values. Essentially manipulating symbolic expressions within a particular context.

*A conceptual issue here is that without parameters, those won’t technically be QFs, just subexpressions. The “basic [QFs]” written in the docs to have “no parameters” actually have default parameter values of 1 or 0, and “adding parameters” really adds function arguments to specify different values.

QFs holding parameter values, default or not, can be represented by callable objects in Julia. Higher order functions can compose them according to any rules. The custom names for unspecific parameters can be done in arguments of a new function, but again it’s not expected to make an arbitrary number of them, but programs generally don’t need to.

Worth mentioning that standard statistical distributions are implemented in Distributions.jl and the QF is accessed by the method quantile.

2 Likes
14

Thank you for looking up my work! Yeah, I did not get started on reconciling my work to Distributions.jl yet. I might have extend that class, because all of my distributions won’t have an explicit CDF or PDF but will always have a close form QF and it’s derivatives (quantile density and possibly convexity), which are not in Distributions.jl

The point estimate for the inverse for these “composed” QFs can be computed by rootfinding, so that’s yet another can of worms (making a generic root finder when the target function is to be specified by the user). Again, done in R, but need to rethink in Julia.

15

Again, easily done using higher-order functions, and you can even use automatic differentiation (e.g. ForwardDiff.jl) to apply Newton-like algorithms. This has already been implemented in Julia by several packages, e.g. Roots.jl or NLsolve.jl or IntervalRootfinding.jl or NonlinearSolve.jl

1 Like
16

That’s what I am here for! Looking forward to embracing the power of Julia ecosystem. Thank you for the tips!

1 Like
17

For those visiting here later, I successfully used Roots.jl for bracketed rootfinding of CDF in the new QDistributions.jl package. Still figuring out the accepted way to do things (eg vectorizing my new quantile densities), but overall I love the experience. What took a lot of gymnastics in R, comes naturally in Julia.

Because quantile distributions can be invented in innumerable quantity, I am thinking of some meta-approach to coding them up (at least simple cases like quantile mixtures). Again, thanks for everyone’s tips and helpful discussions!

1 Like
18

I mentioned this earlier, but I think this could be vastly simplified if you reworked your design, regardless of Julia or R. Currently, you are implementing both the parameters and the single input of (univariate) QFs as function arguments, and the metaprogramming serves to shuttle the arbitrary number of parameters from transforming an arbitrary number of nested QFs to the topmost call. For a handwavy example, if I transform X(p, alpha) and Y(p, beta), I get a function Z(p, alpha, beta).

By contrast, Gilchrist’s writings (as far as I have been able to look up) consistently denote QFs as functions of 1 argument p: general QFs with Q(p), and “basic form” QFs with S(p), which can still have parameters except setting position as 0 and scale as 1. So taking the last example, I transform X(p) and Y(p) into Z(p); the parameters alpha and beta would be stored by X and Y themselves, and basic forms can be represented by storing default values for position and scale. This transformation is much simpler and to my understanding would not require metaprogramming at all, just a consistent set of transformation functions expecting single-input callables. Of course, the tradeoff is that I can’t arbitrarily swap parameter values for the same callable Z, I would have to make an X2, Y2, and Z2. Conceptually this mirrors how quantile functions with different parameters aren’t the same functions and don’t represent the same distributions. I don’t expect this to be an issue in practice.

19

Fixed parameter values slightly defy the purpose, or that’s how I thought, but now that I am exposed to how Distributions.jl is storing parameters inside a struct (and changing parameters means re-initializing the distribution) perhaps I should rethink my R-based approach to melding QFs. It feels like what I want is nest structs and dynamically change the “new()” method. I think density mixtures in Distributions.jl come close to doing that (with single level of nestedness).

With QFs as individual functions (not methods) I agree that fixing parameter values at creation of the function makes the function composition explicit and does not require meta-programming.

Thanks for sharing your thoughts and ideas!

20

Not sure which language you’re referring to here, but that’s not possible for Julia’s new, and I don’t think you need anything like that. I wrote up a toy example of the QF parameters-in-struct+single-argument approach (just in case, the QF parameters are a concept in the use case, they have nothing to do with type parameters in the Julia language).

julia> begin
         linearfunc(x, a, b) = a*x+b # QF parameters in arguments
         struct Linear{T} <: Function
           a::T
           b::T
         end
         Linear() = Linear(1.0, 0.0) # defaults like basic form
         (l::Linear)(x) = linearfunc(x, l.a, l.b) # QF parameters in l
       end

julia> begin
         struct Added{T<:Function, S<:Function} <: Function
           f1::T
           f2::S
         end
         Added{Function}(f1, f2) = Added{Function, Function}(f1, f2) # easier generic parameters
         (a::Added)(x) = a.f1(x) + a.f2(x)
       end

julia> X = Linear() # just showing intermediate outputs
(::Linear{Float64}) (generic function with 1 method)

julia> Y = Linear(2, 5)
(::Linear{Int64}) (generic function with 1 method)

julia> Z = Added(X, Y)
(::Added{Linear{Float64}, Linear{Int64}}) (generic function with 1 method)

julia> Zgen = Added{Function}(X, Y) # slower runtime dispatch but less compilation of many type combos
(::Added{Function, Function}) (generic function with 1 method)

julia> (1.0*3+0.0 + 2*3+5), Z(3), Zgen(3), Added(Linear(), Linear(2, 5))(3)
(14.0, 14.0, 14.0, 14.0)

julia> Added(Linear(-2, -5), Linear(2, 5))(3) # change QF parameters
0

Contrast that with a metaprogramming approach, which I won’t bother implementing but I can at least show you the output expressions:

julia> #= insert metaprogramming project here, pretend it made the following =#

julia> Zfunc(x, a, b) = linearfunc(x, 1.0, 0.0) + linearfunc(x, a, b)
Zfunc (generic function with 1 method)

julia> Zfunc(3, 2, 5)
14.0

julia> Zfunc2(x, a1, b1, a2, b2) = linearfunc(x, a1, b1) + linearfunc(x, a2, b2)
Zfunc2 (generic function with 1 method)

julia> Zfunc2(3, 1.0, 0.0, 2, 5)
14.0

The key difference is that the metaprogramming approach needs to create new functions and methods for every transformation scheme, which irreversibly takes up memory in the runtime and needs to be compiled anew. On the other hand, you could chain the existing constructors in the single-argument approach however you want and reuse the methods; you really only have to compile more for the various numerical types that show up. The user can decide on sticking to inputs of a particular numerical type if they know the operations will ultimately promote to it anyway.

Of course if a user really needs a top-level call to take QF parameter values as arguments, they can still use the single-argument code:

julia> Zfunc3(x, a1, b1, a2, b2) = Added(Linear(a1, b1), Linear(a2, b2))(x)
Zfunc3 (generic function with 1 method)

julia> Zfunc3(3, 1.0, 0.0, 2, 5)
14.0
1 Like