Is there any elegant julian way how to do what python could do with decorator?

General Usage
1

My motivation was to make something like python’s decorator…

But Julia seems to be designed otherwise. Is this intended? For what purpose?

julia> wtf() = show("what the function!");

julia> _wtf = wtf;  # I wanted to remember old behavior

julia> wtf() = show("what?");  # don't call _wtf() here if you don't like stack overflow! 

julia> _wtf()  # but old behavior is gone (is old code still in memory?) 
"what?"

julia> _wtf === wtf  # are there same thing? 
true

julia> _wtf() = show("what?")  # really same? No. 
ERROR: cannot define function _wtf; it already has a value

I thought probably I have not to mix functions and methods so I tried:

julia> collect(methods(_wtf))[1]()
ERROR: MethodError: objects of type Method are not callable

My first impression: “objects of type Method are not callable”?? Really? (OK. Maybe if type is called MethodReflection it could be more understandable)

I really miss some elegant way how to work with functions/methods as objects. Is there any?

2

In general, you probably don’t want to muck about with individual Method objects, these aren’t really intended to be an exposed interface, and their internals frequently change.

What sort of decorator are you trying to write? Depending on what you want to do, you may want to look into Julia’s macros: these can do anything a Python decorator can do, and much, much more since they can appear anywhere (not just at function definition), and have access to the full syntax tree.

3 Likes
3

This works as intended:

julia> wtf = () -> (show("what the function!"); nothing)
#1 (generic function with 1 method)

julia> _wtf = wtf;

julia> wtf = () -> (show("what?"); nothing)
#3 (generic function with 1 method)

julia> _wtf()
"what the function!"
julia> wtf()
"what?"
julia> typeof(wtf)
getfield(, Symbol("##3#4"))

julia> typeof(wtf) === typeof(_wtf)
false

It seems Julia treats anonymous functions differently from normal functions. The long version would be:

julia> wtf = function ()
           show("what the function!");
           return nothing
       end
#1 (generic function with 1 method)

julia> _wtf = wtf;

julia> wtf = function ()
           show("what?");
           return nothing
       end
#3 (generic function with 1 method)

julia> _wtf()
"what the function!"
julia> wtf()
"what?"
1 Like
4

But is it intended? Is it stable? (Is it documented?)

And is it good?

5

No, you get the same behavior if you do foo() = show("what the function!"); wtf = foo.

The difference is that when you do wtf = () -> show(....), wtf isn’t actually the name of the function, it is just a binding that points to the the function. If you then assign wtf = 3, it points to the number 3. Whereas if you do wtf() = show(...), then wtf is a constant referring to the generic function name — it is an error to do wtf = 3 at that point. Subsequent wtf(...) = ... lines add/change methods for the same wtf function rather than changing the binding to point to something else.

11 Likes
6

But if wtf === foo and typeof(wtf) === typeof(foo) why wtf is not just same binding that point to function object?

I don’t look for technical answer (I don’t want to know source and line where it is defined that wtf and foo ). Primary question is why is this designed this way. (Maybe it is a bug?)

7

I guess an analogy to what @stevengj means is the following:

julia> const wtf = [1,2,3]; # similar to defining wtf(...) = ...

julia> _wtf = wtf; # both are now binded to the same thing

julia> wtf .= 100; # similar to adding or overwriting methods of the `wtf` function by redefining it

julia> _wtf # naturally they are still referring to the same thing
3-element Array{Int64,1}:
 100
 100
 100

julia> _wtf === wtf # the same address in memory too
true

On the other hand rebinding wtf with this syntax wtf = () -> (show("what?"); nothing) doesn’t just mutate the methods of the function object, it makes a completely new function and binds the name wtf to it.

A bit more insight:

julia> const f = () -> 1 # similar to f() = 1
(::#1) (generic function with 1 method)

julia> f(x) = 2. # adds another method to function object f
(::#1) (generic function with 2 methods)

julia> f()
1

julia> f(1)
2.0

The only catch here is that functions are isbits and immutable, so the above analogy is probably wrong under the hood but it seems sufficient at a high level.

julia> isbits(f)
true

julia> isimmutable(f)
true
8

I was thinking about answer for SO question OP was saying also about some deprecation phase of function’s keyword arguments.

I was thinking about something like:

f(a,b;new_keyword=7) = show("cool new function")  
if true # deprecation_phase # code here could be more complicated
     _f = f
     function f(a, b; new_keyword=7, deprecated_keywords...)
         something = 7 # elaborate value from new and deprecated keywords
         # _f(a ,b, new_keyword=something) # stack overflow!!! 
     end
end

mohamed82008’s proposal could help avoid stack overflow:

f = (a,b;new_keyword=7) -> show("cool new function")  
if true # deprecation_phase # code here could be more complicated
     _f = f
     f = function (a, b; new_keyword=7, deprecated_keywords...) # this helps
         something = 7 # elaborate value from new and deprecated keywords
         _f(a ,b, new_keyword=something)  
     end
end

Is mohamed82008 's proposal without performance or other penalty?

(Out of topic about access to full syntax tree: could macros rebind local variables?)

Yes. I like to get something like:

f(a,b,newk) = show("cool")

@mapkeywords(f, Dict(:oldk => :newk))

But could macro help with impossibility to rebind function?

9

Yes. But we are talking about our observations, not about language design or intentions. (And we also observe:typeof(foo)===typeof(wtf))

wtf2() = show("new function") # this create function object too. and binding it to wtf2 name
wtf2(x) = 7 # this add new method

# IMHO from language design POV this is doable too:
_wtf2 = () -> show("new function")
addmethod!(_wtf, (x) -> 7)  # why not expose full potential of function object?

So now it seems that we have two possibilities:

  1. wtf() = 7 which is crippling possibility to “decorate”/“copy” code
  2. _wtf = () ->7 which is crippling possibility to add methods.

edit:
3. foo() = 7; __wtf = foo where foo could help to add methods but pollute namespace and worsening readability

Btw:

julia> function a1() foo() end;

julia> @code_llvm a1()

; Function a1
; Location: REPL[33]:1
define i64 @julia_a1_57933() {
top:
  ret i64 7
}

julia> function a2() __wtf() end;

julia> @code_llvm a2()

; Function a2
; Location: REPL[35]:1
define nonnull %jl_value_t addrspace(10)* @japi1_a2_57940(%jl_value_t addrspace(10)*, %jl_value_t addrspace(10)**, i32) #0 {
top:
  %3 = alloca %jl_value_t addrspace(10)*, align 8
  %gcframe1 = alloca [3 x %jl_value_t addrspace(10)*], align 8
  %gcframe1.sub = getelementptr inbounds [3 x %jl_value_t addrspace(10)*], [3 x %jl_value_t addrspace(10)*]* %gcframe1, i64 0, i64 0
  %4 = getelementptr inbounds [3 x %jl_value_t addrspace(10)*], [3 x %jl_value_t addrspace(10)*]* %gcframe1, i64 0, i64 1
  %5 = bitcast %jl_value_t addrspace(10)** %4 to i8*
  call void @llvm.memset.p0i8.i32(i8* %5, i8 0, i32 16, i32 8, i1 false)
  %6 = alloca %jl_value_t addrspace(10)**, align 8
  store volatile %jl_value_t addrspace(10)** %1, %jl_value_t addrspace(10)*** %6, align 8
  %thread_ptr = call i8* asm "movq %fs:0, $0", "=r"() #2
  %ptls_i8 = getelementptr i8, i8* %thread_ptr, i64 -10920
  %7 = bitcast [3 x %jl_value_t addrspace(10)*]* %gcframe1 to i64*
  store i64 2, i64* %7, align 8
  %8 = getelementptr [3 x %jl_value_t addrspace(10)*], [3 x %jl_value_t addrspace(10)*]* %gcframe1, i64 0, i64 1
  %9 = bitcast i8* %ptls_i8 to i64*
  %10 = load i64, i64* %9, align 8
  %11 = bitcast %jl_value_t addrspace(10)** %8 to i64*
  store i64 %10, i64* %11, align 8
  %12 = bitcast i8* %ptls_i8 to %jl_value_t addrspace(10)***
  store %jl_value_t addrspace(10)** %gcframe1.sub, %jl_value_t addrspace(10)*** %12, align 8
  %13 = load i64, i64* inttoptr (i64 140211137321128 to i64*), align 8
  %14 = getelementptr [3 x %jl_value_t addrspace(10)*], [3 x %jl_value_t addrspace(10)*]* %gcframe1, i64 0, i64 2
  %15 = bitcast %jl_value_t addrspace(10)** %14 to i64*
  store i64 %13, i64* %15, align 8
  %16 = bitcast %jl_value_t addrspace(10)** %3 to i64*
  store i64 %13, i64* %16, align 8
  %17 = call nonnull %jl_value_t addrspace(10)* @jl_apply_generic(%jl_value_t addrspace(10)** nonnull %3, i32 1)
  %18 = load i64, i64* %11, align 8
  store i64 %18, i64* %9, align 8
  ret %jl_value_t addrspace(10)* %17
}
10

I think it’s not clear what you are doing all of this for. If your purpose is to do something similar to Python’s decorators, the following code should be enough:

julia> f(a,b;new_keyword=7) = show("cool new function")
f (generic function with 1 method)

julia> @new f(1,1, new_keyword=4, old_keyword=4) # -> macro turns function call to `begin; something = 7; f(1,1,new_keyword=something); end`

Macros take expressions as input and return expressions. The returned expressions are then plugged in the code as if you have written them by hand. So macros are expanded before the code following the macro call, e.g. @new, is even validated, as long as the code is valid Julia “syntax” that can be transformed to a Julia’s abstract syntax tree (AST). This means the code following the macro call can throw errors, overwrite constants, or do anything otherwise invalid, as long as you make sure the returned code/expression by the macro is valid to run.

The main difference between the above macro-based code and a similar Python decorator. to the best I can remember, is that at call site in Python you will just call f, but in Julia you need to add the macro call explicitly. So Python decorators intercept function calls of decorated functions automatically and mutate their behavior. If you want something similar in Julia, you will need a function wrapper: g(x) = @new f(x). I am a bit rusty on decorators so correct me if I said anything wrong.

Alternatively, you can “decorate” the function definition in Julia with a macro call, where the macro defines 2 methods of the function, one with the new keyword only, and another with new and old keywords that basically figures out the arguments to call the first method with and then calls it, i.e. it’s a thin wrapper.

You can also make the macro define an intermediate function with a dummy name and the same functionality as the decorated function definition. The macro can then make another function wrapping the intermediate function with the originally intended function name. So at call site, you can just call f, and it will run the wrapper function which will call the intermediate function. So the possibilities are endless!

2 Likes
11

Exactly!

Python’s flexibility is really nice. Because if you want to write reusable code (Especially if you are package maintainer. But as experienced senior programmer you wanted to do with your private code too) you like to export as simple interface as you could.

If we apply theory of mind we have to see that if I exported function for example plot and user of my package has code like:

plot(Line(x1,x2,y1,y2), size=3)

and then I realized that I need size for something other I like that user don’t need to do anything during deprecation phase:

plot(Line(x1,x2,y1,y2), size=3)
Size is deprecated use thickness  instead!

It would be wrong If I force user to rewrite code immediately to:

@new_version1_0 plot(Line(x1,x2,y1,y2), size=3)

It would be better to break and force to use new keyword.

I hope I explained why I want to just rebind function name.

You could say: Well you could do everything with function body so where is the problem?

There is another level of abstraction. I like to have my package code clean and simple I don’t like to invet wheel twice:

import Deprecator

function plot(x::Line; thickness=1) ... end  # just new functionality

# Nothing wrong happen if I forgot to delete next line after v"1.0"! 
@Deprecator.remap_keywords(my_version<v"1.0", plot(x::Line; thickness=1), Dict(:size => :thickness))

# alternatively remap all methods
@Deprecator.remap_keywords(my_version<v"1.0", plot, Dict(:size => :thickness))

Julia has multiple dispatch and methods. It really complicate matter. But is it really impossible to write something like @Deprecator.remap_keywords ?

12

Yes. But we are talking about our observations, not about language design or intentions.

It is intended, stable, and an unavoidable consequence of how multiple dispatch works – when you define a new method, you are not creating a new function object, you are adding a method to an existing function.

If you want something like a Python decorator, it’s not hard and it doesn’t even require macros:

repeater(f) = function (args...)
    f(args...)
    f(args...)
end

multiply(num1, num2) = println(num1*num2)

julia> repeater(multiply)(2, 3)
6
6

If you want a global binding with the name multiply that you can call instead of repeater(multiply), you can just do

const multiply = repeater() do num1, num2
    println(num1*num2)
end

(That won’t work in the same REPL session because the function bindings are constant.)

If you really want to write this as:

@repeater multiply(num1, num2) = println(num1*num2)

a little bit of macrology can make that happen, but I’ll leave that as an exercise for the reader, as they say.

You may be interested in this SO question and answer:

In short: Julia’s macros, like Lisp’s, are strictly more powerful than Python’s decorators.

9 Likes
13

If you want something like this, that’s trivial, just define a new method with size keyword that prints the warning and then calls the correct method with thickness keyword. You will probably need to make it more generic though using kwargs... and then just iterate over kwargs and print all the necessary warnings. Then call the correct method with the new keyword arguments. I don’t see how this has anything to do with macros or decorators to be honest.

1 Like
14

Trivial like this? → 

julia> plo(a,b; size=1) = 1;

julia> plo(a,b; thickness=1) = 2;

julia> plo(1,2)
2

julia> plo(1,2, size=2)
ERROR: MethodError: no method matching plo(::Int64, ::Int64; size=2)
...

And it is not my problem to solve! BTW if it is so simple - you could write answer to SO. :slight_smile: I tried and it was way more complicated than it seems at first sight. Here is my experiment:

julia> depre = true
       real_f(a,b;newk1=3,newk2=4) = show((a, b, newk1, newk2))
       type MySingleton end
       novalue = MySingleton()
       function depre_f(a, b; newk1=novalue, newk2=novalue, deprecated...)
           dic = Dict(deprecated)
           k1 = k2 = novalue
           if :oldk1 in keys(dic) k1 = dic[:oldk1] end
           if :oldk2 in keys(dic) k2 = dic[:oldk2] end
           if newk1!==novalue k1 = newk1 end
           if newk2!==novalue k2 = newk2 end
           real_f(a, b, newk1=(!==(k1,novalue) ? k1 : 3), newk2=(!==(k2,novalue) ? k2 : 4))
      end
      f = depre_f 
  else
      f = real_f
  end

I know - it is really ugly - nobody wants to do it this way (me neither).

I don’t see why you don’t see that problem from SO is OP’s problem and my question about possibility to remeber old function object and rebind function name to new function object (aka decorator) is another question.

You could see motivation for my question but it looks like I had to avoid it as too big distraction :slight_smile:

15

Nope, trivial like this:

julia> function plo(a,b; thickness=1, size = nothing)
           if size ≠ nothing
               warn("size is deprecated, use thickness")
               thickness = size
           end
           println("thickness is $thickness")
       end
plo (generic function with 1 method)

julia> plo(1, 2, size = 9)
WARNING: size is deprecated, use thickness
thickness is 9

Perhaps you should read up on functions.

(Also, Base has some fancier deprecation framework, but I wanted to keep it simple.)

1 Like
16

Thanks.

17

Are you proposing higher priority to old keyword than to new one?

(BTW. from SO question we don’t know if nothing is legal value for keyword. EDIT: And mapping has also more complicated: k1, kw1, key1, keyword1 for same class)

18

(BTW. from SO question we don’t know if nothing is legal value for keyword)

Then just use the other option:

function plo(a,b; thickness=1, deprecated_kwargs...)
    kwdict = Dict(deprecated_kwargs)
    haskey(kwdict, :size) && warn("....")
    thickness = get(kwdict, :size, thickness)
    ....
end
2 Likes
19

I like missing value trick but keyword priority problem remains…

20

I don’t understand what you mean here.

Regarding the other point: I was merely pointing out how to do what @mohamed82008 suggested because it seemed that you needed help with it. Modifying this for a more complex example would require some effort on the part of the reader, but the principle is the same.

The SO question in particular is mixing two things: deprecation (mentioned in a comment later on) and “cascading” keywords. I think that for deprecation, it is best to inspect the value in the function body.

Finally, if you are concerned about sentinel values, you can capture all keyword arguments in eg args... and examine them as is.

1 Like