Expensive Code Generation for Functions for Reuse

General Usage
1

Hello. I’m looking for some guidance on a design decision to make. I have an algorithm which makes use of 10 to 20 specialized function that are called repeatedly, however, none of the functions are known until the necessary info is supplied at runtime. One strategy is to generate the code and cache the functions for later which is what the existing codebase does. Generating the AST for the function requires running expensive steps (i.e. graph algorithms), so I am unsure if there is even an alternative route besides eval-ing and storing the functions for reuse. Moreover I don’t think I can take a memoization route since the procedures are inherently stochastic. I have a couple of questions

  • In similar scenarios where the functions ought to be stored somehow, what do people usually do? Do you do as I explained above, or something better? Are there example packages that run into similar design choices?
  • Would generated functions work here? I also do not wish to pollute the namespace, so it seems I need anonymous functions. Is eval the go-to here? evaling these functions sounds like a type unstable mess :sweat_smile:.
  • How could I make sensible stack traces for the user if the code is generated?
2

Can you give an example of what the functions you need actually look like? Are you sure there’s no way a normal function can’t handle it? Seeing examples will probably help.

2 Likes
3

Pass the functions as arguments to your algorithm?

If you’re passing 10–20 functions to your algorithm all at once, maybe those should come from a type rather than a single argument? e.g. if you have an argument x::T, then you can functions foo(x, …) that the caller can overload for a type T to do different things.

It’s hard to say more without knowing more specifics.

1 Like
4

It’s impossible to give advice specific to your application, given that you didn’t provide code, but the general rule of “avoid metaprogramming” is always valid. In Julia it’s possible to move values into the type domain to achieve compilation at run time, so metaprogramming is only rarely necessary or desirable.