Fighting latency and the future of FunctionWrappers

In our constant battle to reduce time-to-first-X, I’m finding it very, very valuable to use the precompile approach of my bottleneck functions (diagnosed with the amazing SnoopCompile frameworks, thanks a bunch for that!). Since it is only possible to precompile a function for a finite set of input types, a crucial part of my strategy is to put a FunctionWrapper barrier between the user-facing API functions (that can take wildly varying types as input), and the digested, internal functions, that do the heavy lifting. The latter will have a predictible and small set of possible input types, so it can be precompiled to great effect. This is a toy example of the approach

using FunctionWrappers: FunctionWrapper

struct DigestedInput
    test::FunctionWrapper{Bool,Tuple{Int}}
end

digest(xs) = DigestedInput(x -> test(x, xs))

test(x::Int, xs) = any(test.(x, xs))
test(x::Int, x´::Int) = x == x´

myfunc(x, input) = myfunc(x, digest(input))

function myfunc(x::Int, d::DigestedInput)
    # heavy function using d.test
    result = d.test(x)
    return result
end

precompile(myfunc, (Int, DigestedInput))

Then, a user call with a complicated input type, such as myfunc(0, (1, (2, 3), ((4, 5), 6), (x for x in [2,0,4]))) will quickly be forwarded to the precompiled method myfunct(::Int, ::DigestedInput), where the heavy lifting is made.

In my work I have found this to be a good way to reduce latency while allowing expressive type flexibility in the user API.

But now comes the problem: the basic pillar of this approach is this very special package by @yuyichao called FunctionWrappers that does some inscrutable magic with pointers and whatnot. Way beyond what I understand in any case. In my usecase it essentially encapsulates operations on arbitrary types, to produce a result of known type.

The package is quite old, but it is not registered and is still labeled as experimental, I believe. I assume it is because there are some lingering limitations, e.g. it cannot wrap functions with kwargs, as far as I understand. This situation is quite uncomfortable as FunctionWrappers provides a crucial functionality that is not available in any other way in Julia. For me at least its role in the battle to reduce latency is more important than ever.

Recently FunctionWrappers broke for me in v1.8 beta1. Here is the Github issue. The problem is that I am at a loss to try and help resolve it, because it requires some deep understanding of Julia internals.

I hope the above has made a case for my plea: would it be possible to incorporate FunctionWrappers as first-class mechanism in Julia, or at least put it into the stdlib? I keep basing my work on it, but its current status makes me quite uneasy about it.

(Of course, if somebody can suggest a better/more official way to do what I explain above, I would be very grateful!)

2 Likes

I’ve been fighting with it this week with some help from @tim.holy and @Elrod . Right now I’m a bit discouraged since it doesn’t seem to effect precompilation like this. You can see all of this at:

My hope is to have a version of DiffEq soon where it can auto-wrap via a FunctionWrapper and effectively treat it as “C solver mode”, i.e. just a precompiled ODE solver which it sticks a function pointer into, like is done for the Sundials.jl .so. But FunctionWrapper’s lack of specialization seems to trip up precompilation no matter what I do and just recompile the solver anyways.

Anyways, I haven’t tried everything here though so fingers crossed that PR can actually work :sweat_smile: .

2 Likes

Oh, what a coincidence! :smile:

So, as far as I understand, your issue is that the wrapped function is not precompiled. In my case that is usually a simple translation layer. The big computation is outside the wrapper, so I don’t have that problem. But I was not aware of this limitation. Do you already know if it is solvable?

Regarding my issue with FunctionWrappers in julia v1.8 (which makes FunctionWrappers fails very generally, whenever the argument type wraps a heap reference), I’m trying to bisect in Julia to see when the breaking change was made. I’m not even sure it is an issue with FunctionWrappers.jl or rather an unintended breaking change in Julia itself.

Not even 5 hours after posting the FunctionWrappers issue in Github, @yuyichao has already fixed it. I’m floored, and a little ashamed of my doubts :-D. It is clear that the package is maintained at a level beyond first class, even despite its experimental status. Many, many thanks @yuyichao!

5 Likes

@ChrisRackauckas, does that help?

Nope :cry:. Didn’t budge the compilation time, still recompiles the full solver package.