How can I capture the current definition of a method in order to be able to invoke it even if it is later redefined ? That is, how to defeat both the standard method redefinition and Revise ?
I’ve looked at the doc, googled it, and searched here for this… it seems I’m missing the right keywords.
julia> f() = 3
f (generic function with 1 method)
julia> Base.get_world_counter()
0x0000000000007a08
julia> f() = 5
f (generic function with 1 method)
julia> Base.invoke_in_world(0x0000000000007a08, f)
3
This is that!
Why? For “security” reason to ensure I can call the Base method I’m expecting to call. Also to use delegate part of a redefinition to the previous specific definition.
Slow: I get this (I’m new to julia but a compiler guy). I don’t get why I’m in the wrong place w.r.t. security ? When the runtime start I can capture some methods, I can be certain to have “first hand” if my module init is in a sysimage. Then I’m protected against any redefinition of the method ? Is there as way to bypass this ? (assuming Base.invoke_in_world itself is protected likewise and excluding low-level stuff like direct llvm/asm generation ?
(note: this is not a question of correction but of security, as in protection against adversarial behavior of some code loaded after mine. The scare quotes on “security”… hum… because there so much you can do with such a dynamic semantics)
This use case is dual of 1). I don’t get your reply : if I want to override some method of, say, Base.foo but Base.foo is big and does of lot of work and I only need to change a subpath, then I must redefine Base.foo, no ? It is still useful (“safer”) to delegate the larger part of the work to the original Base.foo, and thus inherit any future improvements ?
(I’m from the FP world, but, sometime, only sometime, virtual based inheritance is convenient )
what are you trying to protect against? The methods you “captured” is useless if user don’t also do invoke_in_world when they are not calling your public-facing function. And say they are calling your function, what’s stopping a later package from changing your outer-most functions’ methods?
you understand they could just override your whole module right?
use a internal function to do the un-changing part of your function. Don’t dynamically time-tarvel with invoke, compiler can’t optimize that.
In the end you can do whatever you want, but know that the package produced with this recipe will be hard to integrate into the ecosystem and likely slow (if performance matters)
hum… thank for your reply but I still don’t really get it
Don’t dynamically time-tarvel with invoke, compiler can’t optimize that.
I get it… written a few compilers in a previous life
use a internal function to do the un-changing part of your function.
it is not unchanging, it is un-changed. They there is an exported, public API method in Base. Let’s call it Base.foo(…). It is 100 lines long. I want to have, say, a custom REPL experience. The REPL do call Base.foo(…). I can override it to customize. But I need only to change a tiny part of Base.foo(…), say a little case depending on some input value. I think it is still better if by override of Base.foo(…) do delegate to the original implementation. (actually I don’t see any other solution but to copy/past the full origin code of Base.foo(…))
if your new foo rely on old foo, just call the old foo → _foo. This is a common pattern, where the foo is high level / flexible, _foo is directly doing the fixed set of heavy lifting.
you understand they could just override your whole module right?
Yeah… but for scenario 1) I’m in the sysimage. Otherwise I couldn’t do anything meaningful anyway. I don’t care what the code calling my code will do. I just want strongest guarantee that if my code do run, then it does what it was designed to do. E.g. equivalently, all calls from my code could be inlined, full transitive closure downtown Core! (bad idea, but this would be equivalent to what I tried to describe in point 1)
if your new foo rely on old foo , just call the old foo → _foo . This is a common pattern, where the foo is high level / flexible, _foo is directly doing the fixed set of heavy lifting.
I’m afraid I’m missing something important here…
The old foo is in Base (the actual Base of the Julia distribution).
How can I do this foo → _foo ? Really I don’t get it…
thank you… So I’m not mad invoke_in_world is indeed what I was looking for… and the only possible way given the fully dynamic nature of the overrides (with back-link and code re-generation).
Question: why is it not in the doc ? I understand that users could abuse it… But the use cas defined in 2) above seems… well useful ?
the reason is that usually you want to use the latest methods, everywhere… if somehow X is designed to be customizable but you can’t customize it without doing this hack, then X is poorly designed.
Because “world age” is an implementation detail to achieve compiled functions in the context of a dynamic programming language with eval.
You don’t have to use invoke_in_world for that.
@eval Base function foo()
[.. old definition of foo]
new_var = 5
do_with_func(new_var)
[... old definition of foo]
end
Getting the old definition itself can be done by looking up its source code, parsing the code and manipulating the resulting AST (though of course you have no guarantee that the source on disk is the same one that was used to compile the original function in the first place).
That aside, Cassette.jl is one tool doing something like this, but it comes with HUGE compiler and performance drawbacks and on top of that it’s a very brittle tool for all the right reasons.
I’ve looked at some of the rewriting tools (Cassette, macro + MacroTools or MLStyle, what was done in the context of the source-to-source autodiff, etc…). It’s very nice.
But it’s heavy for a little thing like…
# My Code
function Base.repl_cmd(cmd, out)
if cmd.exec == ["foo-bar"]
# special handling
else
orginal_base_repl_cmd(cmd, out)
end
end
after thinking a tiny bit about it… it is not clear what semantics orginal_base_repl_cmd could have without a global world age. A chain of method re-definition (a “function age” for given fixed symbol) seems tricky with the specialization… Could maybe be defined for monomorphic functions… anyway… thanks for all the info.