How to generate a customized `for` loop code?

General Usage
, ,
1

I’m using the macro @hyperopt from package hyperopt.jl to optimize some hyperparameters for a machine learning model. However, I’m pretty constrained by the syntax of @hyperopt, which is designed to be used as follows:

hoRes = @hyperopt for i=numOfSamples, sampler=someSampler,
    hPar1 = valRange1, 
    hPar2 = valRange2,
    ...
    hParN = valRangeN
    
    modelFunc([hPar1, ..., hParN])
end

I want to wrap the above code in a function that can take a non-fixed number (N) of hyperparameters as a Vector argument as I’m testing many different variants of my model. However, due to the macro formalism, I cannot find a straightforward way to achieve this without explicitly writing out the N functions, each corresponding to optimizing a specific number (1 <= I <= N) of hyperparameters.

I believe there is a way to write a generated function (or a macro), genFunc, that directly modifies the code for the for loop when I specify the number of hyperparameters as input. It probably should look like something as below:

function wrapperOpt(::Val{N}, pars, valRanges, numOfSamples, someSampler) where {N}
    # N == length(pars) == length(valPranges)
    f = genFunc(Val(N), modelFunc, numOfSamples, someSampler)
    f(pars, valRanges)
end

where genFunc modifies the code involving @hyperopt and the required for loop.

However, there seems not to be an example of how to generate a code of for loop in the official documentation (especially with multiple condition expressions). And I don’t have much experience with metaprogramming either. If someone could help me out, I would much appreciate it! Thank you!

2

Macros modify expressions when parsed from the text, before they’re evaluated. Since you’re passing in an evaluated type parameter with a value, macros work too soon. With much more limitations, @generated functions work on input types and generate the function body expression. That is plausible. It won’t work like your genFunc, a @generated function is the callable; it makes its own code body expression, not another callable. So you’d want to pass in pars and valRanges so the @hyperopt loop expression has something to reference. That’s about as far as I can surmise because without providing specific examples of pars, valRanges, numOfSamples, someSample, no one else can begin on crafting the expression. It’s also not certain from HyperOpt.jl’s documentation whether it’s possible to do something like hPar2 = valRanges[2] accessing valRanges from outside the loop, all the examples seem to create their ranges inside the loop. Hopefully it is, because it’s in maintenance mode and can’t change in the foreseeable future.

Some quick tips on working on expressions since you’re doing the work:

  1. To access source code as an expression, paste it in x = quote ... end to make an Expr instance; the assignment is just to keep it live. Don’t worry about pasting macro calls, they’re not expanded yet at the early phase of parsing where Expr are made.
  2. Normally when an Expr is printed, it’ll try to show its source code appearance. To see its more direct abstract syntax tree (AST) structure, use dump(x). Note that dump has to enter the tree recursively and has a default depth of 8; if that’s not enough to reveal some important parts of the tree (you’ll see Expr nodes), change it dump(x; maxdepth=12).
  3. Push comes to shove, you can insert code (Symbol and Expr) into an Expr by push!ing to some branch of the tree because it’s built out of Vector{Any}. When you can however, work by $-interpolating into a quote end block that looks more like the source code, it’s much easier to read and edit.
1 Like
3

Thanks for the reply!

I forgot that the @generated function does not support closure, so we can’t write an anonymous function inside an Expr to be generated when it is first called with a specific Val(N).

However, I don’t understand why we cannot design a macro to automate the process of generating functions like the following without knowing the types of x and y:

function f2(x, y)
    @hyperopt for i=1:2, sampler=y, 
        x1 = x[1],
        x2 = x[2]
        model([x1, x2])
    end
end

function f3(x, y)
    @hyperopt for i=1:2, sampler=y, 
        x1 = x[1],
        x2 = x[2],
        x3 = x[3]
        model([x1, x2, x3])
    end
end

As far as I understand, we just need to know how to construct a for loop expression without explicitly writing it out but using some functions to build the sub-expressions and concatenate them. I found a possibly relevant thread about concatenating assignment expressions, but I don’t know how to make them part of the for loop expression. I think if I use the dump function you mentioned to analyze the expression structure of a for loop, then I might be able to do it.

If you have better ideas, I would much appreciate it! FYI, an example of those variables are:

varRanges = LinRange.([0:0.01:1, 0:0.02:2, 0:0.03:6]) # the values (ranges) for all the parameters
numOfSamples = 10 # to be assigned to `i` in the for loop
someSample = Hyperopt.RandomSampler() # The sampler provided by Hyperopt.jl

We actually don’t need pars if we can name the hyperparameters inside the macro before for loop expression. e.g., construct pars = [Symbol(:a, j) for j=1:N], then interpolate its element values as the variable names for the assignments in the expression.

4

Yep, can’t make functions in generated function bodies.

The types of x and y do appear irrelevant in this case. It’s mostly the Val(N) part that can’t have a value in the phase when a macro works, and you really need that N value to craft the method body. A normal function however can have a value for Val(N) at runtime and use it to craft an expression, which you can eval in the global scope. Maybe that’s feasible too.

Thank you for the examples, so let me ask this in return to check if it’s feasible for @hyperopt’s transformed expression: does this code run properly? If you paste this into a trivial @generated function with no processing, does the function run properly without hitting some @generated limitations?

# this would be what genFunc does
# hopefully as arguments
varRanges = LinRange.([0:0.01:1, 0:0.02:2, 0:0.03:6])
# these two could be baked in via Val
numOfSamples = 10
someSample = Hyperopt.RandomSampler()

# quote
  @hyperopt for i=1:numOfSamples, sampler=someSample,
    a1 = varRanges[1],
    a2 = varRanges[2],
    a3 = varRanges[3]
      model([a1, a2, a3])
  end

PS format those function examples with these

```
code here
```
5

Did you mean something like this?

model(var) = sum(@. var[1] + (var[2]-3)^2+ (var[3]-100)^2)

@generated function f()
    varRanges = LinRange.([0:0.01:1, 0:0.02:2, 0:0.03:6])
    numOfSamples = 10
    someSample = Hyperopt.RandomSampler()
    quote
        @hyperopt for i=1:numOfSamples, sampler=someSample,
            a1 = varRanges[1],
            a2 = varRanges[2],
            a3 = varRanges[3]
            model([a1, a2, a3])
        end
    end
end

Unfortunately, it did not work:

ERROR: The function body AST defined by this @generated function is not pure. This likely means it contains a closure, a comprehension or a generator.

P.S. Thanks for the reminder! I knew the usage of ``` but forgot.

6

Shame, nothing else in the returned Expr has those, so it’s @hyperopt doing something.

We’ve gone over how we can’t have the necessary information at parse-time (and your input isn’t source code anyway), and now that we know @hyperopt does things that compile-time @generated functions can’t do, the remaining option is a runtime eval of an expression returned by a function. Since runtime is the latest phase, there’s a lot of things that can’t be changed, but at the same time we have more information.

function genFuncExpr(N)
  pars = [Symbol(:a, j) for j=1:N]
  funcexpr = quote
    function (varRanges, numOfSamples, someSampler) # anonymous
      n = length(varRanges)
      if n != $N  error("Takes "*string($N)*" ranges, "*string(n)*" provided")  end
      @hyperopt for i=1:numOfSamples, sampler=someSampler
        model([$(pars...)])
      end
    end
  end
  # don't know a way to interpolate to for header, so direct Expr mutation
  # bear in mind this bakes in the varRange values, not expressions.
  forassigns = funcexpr.args[end].args[end].args[end].args[end].args[begin].args
  append!(forassigns, ( :( $(pars[j]) = varRanges[$j] ) for j in 1:N))
  funcexpr
end

# could be const f for type stability
# to see the resulting expression, don't do the eval yet
f = eval(genFuncExpr(3))
f(LinRange.([0:0.01:1, 0:0.02:2, 0:0.03:6]), 10, Hyperopt.RandomSampler())

That seems to do what you intended, but again, not sure if @hyperopt can accommodate this, so you should check if f works.

7

Something about this whole approach feels off to me, trying to do complicated metaprogramming to work around the limitations of some other complicated metaprogramming. I would take a step back and investigate what @hyperopt is doing and whether it contains building blocks that can be reused for the desired generalizations.

1 Like