julia> f(x::Int) = "Int"
f(x::Float64) = "Float64"
@generated function g(x)
if hasmethod(f, Tuple{x})
return :(f(x))
else
return "???"
end
end;
julia> g(1)
"Int"
julia> g(1.0)
"Float64"
julia> g("a")
"???"
… but the following more clumsy way of writing an equivalent generated function throws an error:
julia> @generated function h(x)
if length(methods(f, Tuple{x})) > 0
return :(f(x))
else
return "???"
end
end;
julia> h(1)
ERROR: code reflection cannot be used from generated functions
Does that mean that the initial g(x) is also invalid but just misses an explicit sanity check?
Also, what exactly does “invalid” mean here? I’m aware of the problem that the generated functions do not get updated as more methods are added to f(x) and I’m okay with that. Is that the only surprise I should be expecting, or can either of these generated functions corrupt the entire Julia runtime? If the answer to the last question is yes, then what about the following workaround?
julia> hasmethod_f(_) = false
hasmethod_f(::Type{Int}) = true
hasmethod_f(::Type{Float64}) = true;
julia> @generated function i(x)
if hasmethod_f(x)
return :(f(x))
else
return "???"
end
end;
Edit: I’m basically repeating the question from Can I use hasmethod inside @generated function?. Maybe things have evolved in the meantime, and maybe someone could give a little more detail?
Generated functions must not mutate or observe any non-constant global state (including, for example, IO, locks, non-local dictionaries, or using hasmethod).
It’s worth asking yourself if you really need a generated function though. In recent Julia versions, hasmethod can be inferred statically by the compiler, so in almost all cases, you can just use a regular if statement without @generated. Even if you really need to use @generated you can call hasmethod outside the generated function and wrap it in Val, e.g.
cond = hasmethod(f, Tuple{T})
my_generated_function(Val(cond))
[...]
@generated function my_generated_function(::Val{cond}) where {cond}
if cond
...
else
...
end
end
For context, I’m currently working on a coordinate conversion package. The idea is, given a tuple of coordinates c (e.g. Cartesian coordinates c = (x,y,z)) and a target system s (e.g. s = Longitude()), I want to test whether there is a conversion method from any subset of c to s. So in the above example, I want to test whether I can convert from Cartesian x to Longitude() (I can’t), from Cartesian y to Longitude() (I also can’t), and so on, until eventually I’ll test (x,y) -> Longitude(), and that will work. If I find such a conversion method, then I want to use it, if I can’t then I want to try other manipulations on s and c and repeat the conversion test until eventually I find a conversion that works.
I’m aware that Julia is getting better and better at moving seemingly runtimey things to compile time without a human explicitly telling it to do so, but in my experience the combinatorial explosion of checks that I want to perform is at the brink of what Julia is capable of inferring statically. Static inference is absolutely crucial for my application, so inference working only sometimes was quickly becoming very frustrating. By contrast, @generated gives me explicit control over what happens at runtime and compile-time, and hence this feels like the more sane choice.
Thanks for the pointer! Unfortunately, I’m struggling quite a bit with that PR, though, so I’m not sure how to adapt it to my situation. Also, I want to ask once again, what exactly does “well formed” mean here? To my understanding, the main concern of the PR seems to be to add backedges from the generated function to all the methods that could invalidate it so that recompilation is triggered whenever new methods are added or existing ones are edited. I’m okay with giving up on this dynamic recompilation, so if that’s the only concern then I believe using hasmethod() should be fine for my use case. The only thing I want to 100% rule out is the potential runtime corruption alluded to in the documentation.
Yes, that was what I was going to suggest. The following worked for me:
julia> using Combinatorics
julia> @generated function f(x::Tuple)
ex = :(error("no applicable method found"))
for i in reverse(collect(combinations(1:length(x.parameters))))
ex = :(if Core._hasmethod(Tuple{typeof(foo), $(x.parameters[i]...)})
return foo($(Any[:(x[$i]) for i in i]...))
else
$ex
end)
end
return ex
end
f (generic function with 1 method)
julia> foo(::Int, ::Float64) = "foo"
foo (generic function with 1 method)
julia> f((1, 0x02, 3.0))
"foo"
julia> @code_typed f((1, 0x02, 3.0))
CodeInfo(
1 ─ nothing::Nothing
│ nothing::Nothing
│ nothing::Nothing
│ nothing::Nothing
│ nothing::Nothing
│ nothing::Nothing
└── return "foo"
) => String
Note I did end up using Core._hasmethod directly, otherwise there were some error checks when constructing tuple types that didn’t get optimized away fully
Runtime corruption or not “inference on the generated function may be run at any time”. It’s simply unknowable if this will do what you intend because you can’t know that this will be first inferred after the definitions you want to introspect exist. Which then, yeah, will lead to runtime corruption somewhere.
That worked like a charm, thanks a ton! And in particular thanks for the tip regarding Core._hasmethod(). Don’t think I would have figured that one out on my own.
I don’t get the difference between calling f(x) and hasmethod(f, Tuple{typeof(x)}). If I can’t be sure that some methods exist, then function calls should be equally malformed, no?
If I understand correctly, then this paragraph from the documention should be relevant here:
Some operations that should not be attempted include:
[…]
5. Calling any function that is defined after the body of the generated function. This condition is relaxed for incrementally-loaded precompiled modules to allow calling any function in the module.
What I was saying is that this level of method resolution would have been good enough for me. I don’t need my function to be extendable by downstream packages. Just dispatching based on the methods available in the current package would be good enough for me.
I dunno, if there’s one thing I’ve learned about compilers it’s that breaking their assumptions will almost always break your assumptions about what it’ll do. And if it doesn’t right now, it almost surely will in the future. I’ve not thought deeply about this particular case for precisely that reason.
Fair, but the main point I’m drilling down on here is what exactly are the compiler’s assumptions? (I also do digress into speculations on what might happen if I violate the assumption, but that’s only to provide context. Feel free to ignore those parts.) Specifically, my question is: is it really true that calling f(x) is perfectly fine but calling hasmethod(f, Tuple{typeof(x)}) isn’t?
I’m not sure either is perfectly fine. Recall that a generated function is restricted to the world age during its definition so it doesn’t see any further f methods. You said you were fine with that from the start, but that adds a significant maintenance difficulty that ideally isn’t there i.e. we can add f(x::Float32) and g(1.2f0) would work even if it compiled to ??? previously. hasmethod(f, ...) is in a worse position because it doesn’t have that frozen world age by default on top of being called at compile-time, and it actually works better in a plain function now.
It’s not clear how much metaprogramming you need for coordinate conversion, but if your intent is to move as many calls and branches to compile-time as is reasonable and especially if you’re dealing with world age, then generated functions are actually not the ideal tool. That’s what Tricks.jl is hacking around. It’s just not specified what “reflection” of the global method and binding tables could occur at compile-time and be tracked alongside plain calls and variable usage (Tricks.jl said that hasmethod is in 1.10+, but that’s not a documented language feature), let alone more complicated reflection like this.
These days it is possible to read the method table in a generated function, but to do this correctly you need to:
Query it at the expansion age
Compute world bounds for any property derived therefrom that you’re using
Add appropriate edges to the CodeInfos edge list
There isn’t really any helper code to help you do 2-3 correctly, so it’s not really encouraged. Putting the hasmethod into the generated body as suggested above is a good solution because inference computes all of those things correctly automatically.