Announcing Traits.jl - a revival of Julia traits

[schlichtanders]

Dear Julia community,

Today I want to announce a package of mine, GitHub - schlichtanders/Traits.jl: traits for julia: dispatch on whatever you want

After more than a year of intense experimenting with traits in Julia, I finally feel happy with this version to announce it publicly. Please ignore the little warnings which pop up when installing the package they will be fixed soon.

Check out the git repository where you can also find the installation instructions (it is currently available via a custom registry of mine)

I am glad about every feedback.
Thanks,
Stephan Sahm aka schlichtanders

EDIT:
I added some extra documentation:

• some standard traits functions coming with Traits.jl GitHub - schlichtanders/Traits.jl: traits for julia: dispatch on whatever you want
• an extensive example on how to dispatch on whether functions are defined for certain types or not with the help of Traits.@traits and IsDef.jl
  GitHub - schlichtanders/Traits.jl: traits for julia: dispatch on whatever you want

[schlichtanders]

thanks tshort for reporting that the last example for comparison to SimpleTraits.jl was wrong

I had a testing test/README.jl which I updated, however forgot to update the front README.md. That is now fixed.

Please enjoy the updated README part: GitHub - schlichtanders/Traits.jl: traits for julia: dispatch on whatever you want

[visr]

Would you mind explaining a little bit more on what the advantage of this method is over SimpleTraits.jl? From the readme it only says:

As you see, this package recommends to use julia’s standard dispatch mechanism instead of custom macro definitions. This way the interactions with normal, non-trait-based dispatch becomes way more natural.

And how does this approach compare to Traitor.jl? Or do you qualify for a beer from @tim.holy, as in I will bet one beer... · Issue #8 · andyferris/Traitor.jl · GitHub.

[Mason]

Very interesting package! I like that you allow the user to define traits which may dynamically dispatch, but I think you should put a warning somewhere in the readme that if the user uses a trait pattern that depends on runtime information, they will generate slow code.

Can I ask why this is on a separate registry from the general registry? Is that just a stopgap until you can register your dependant packages or a long term solution for you?

[schlichtanders]

Hi @visr. Thanks for your question

Here some advantages of Traits.@traits over SimpleTraits.jl

1. (this is the most important advantage from my point of view)
   no need for @traitimpl or in general a separation between a standard julia function and a Trait

2. the @traits functions are kind of compatible with existing standard julia functions in that they only overwrite the respective standard dispatch part. I.e.

   @traits f(a::Vector{Int}) = true
   # or more complex
   @traits f(a::Vector{Int}) where {isSometrait(a), isAnothertrait(a)} = true
   

   really only overwrite the previous definition of f(::Vector{Int}), as usual in Julia.

3. Bool functions are just one concrete case of what @traits supports (it seamlessly also works with multi-value traits like defined in the standard Julia Iterator interface Base.IteratorSize)

4. it works with multiple traits at once, in a transparent manner. Concretely if you dispatch on say both isTrait1 and isTrait2 like

   @traits f(a) where {isTrait1(a)} = "first"
   @traits f(a) where {isTrait2(a)} = "second"
   

   and have an a for which both isTrait1 and isTrait2 are true, you will be prompted to fill the ambiguity as usual, which can be solved by defining

   @traits f(a) where {isTrait1(a), isTrait2(a)} = "both"
   

5. also traits with two or more arguments are naturally supported. E.g. if areinorder(a1, a2, a3, ...) checks whether a1 <= a2 <= a3 ..., this just works

   @traits f(a, b, c) where {areinorder(a, b, c)} = true
   

6. the implementation of @traits is very easy to explain on high-level. It just splits up the @traits function into an outer and an inner function, the outer doing the standard julia dispatch, and the inner doing all the extra dispatch functionalities.

7. a small syntax-nice-to-have: You can freely mix normal where syntax with the new syntax. No need for extra character ; to separate them, but the implementation is smart enough to distinguish the different functionalities on its own.

8. another syntax-nice-to-have: You do not need to have explicit TypeVariables like

   @traits f(a::T) where {T, eltype(T)::String} = "something"
   

   but if your function is similar to eltype in that if it gets a value, it returns eltype(typeof(value)) you can simply do

   @traits f(a) where {eltype(a)::String} = "something
   

There probably a couple of disadvantages, however I haven’t really used SimpleTraits.jl and hence cannot tell too much. One obvious point is that so far no Traitor.jl like syntax is supported.

---

Concerning your second question, how this approach compares to Traitor.jl, I hope that some points from above already clarify differences.

• One crucial aspect is again, that Traitor.jl relies on a special convention about Traits types.
  Concretely, in Traitor.jl you have to define a proper type-hierarchy for your traits-types where the abstract type constructor is overloaded respectively - if I understood it correctly. With Traits.@traits nothing like that is needed, it just works with whatever function you have, e.g. Base.IteratorSize.
• Another thing is the syntax: I prefer using Julia where, while Traitor.jl syntax was developed before there even was a Julia where, which back then was the best syntax available, but today feels a bit overloaded and unintuitive for me.
• last but not least, the Traits.@traits syntax is already now pleasantly stable and I guess with only waiting until JuliaCon it is soon going to be production-level stable

I hope that can clarify some of your questions. And open some others

[schlichtanders]

Thanks @Mason. SimpleTraits provides a macro to check whether the traits-functionality generates optimal code. For sure, something like this (if not the very same function) can be included here as well.
EDIT: I will also include a warning into the README

@Mason

Can I ask why this is on a separate registry from the general registry? Is that just a stopgap until you can register your dependant packages or a long term solution for you?

I definitely want to register the packages within General. It is just, that I spend a lot of time, cleanly separating different functionalities I needed into different sub-packages. Feel free to explore the dependency list! They are soon to be announced on their own ;-). As the concrete splitting was not stable for a long time, I used my private registry which solved the problem for me nicely.

[tim.holy]

I like the piggy-backing on where even better than the double-:: (which @schlichtanders impressively intuited was indeed the best syntax at the time).

WRT whether this is the “beer”: the beer is really for GitHub - andyferris/Traitor.jl: An early prototype of a traits system for Julia, which at the time seemed like the one missing piece of Traitor. From a brief glance I don’t think that’s in schlichanders’s Traits.jl. But I’m not sure I’m as much of a fan of that functionality now; I’m beginning to think perhaps it should be up to the owner of the function to decide if there should be a trait-dispatchable method, and if so to write the “depot” method in the package that owns it. I’ve even wondered if we want a way of marking arguments as being non-specializable from the standpoint of method ordering.

In terms of the differences from Traitor, I think in many cases the type hierachy is not a negative, it’s something you want. I’ll note that the given example of Base.IteratorSize was consciously designed as a type hierarchy in exactly the manner illustrated in Traitor:

julia> subtypes(Base.IteratorSize)
4-element Array{Any,1}:
 Base.HasLength  
 Base.HasShape   
 Base.IsInfinite 
 Base.SizeUnknown

Nevertheless, updating the syntax and generalizing to any computed function (hinted at also in one of the final sections of the Traitor.jl README) is a good thing.

[schlichtanders]

Thanks @tim.holy
to confirm, I haven’t solved the betray macro. If I understood it right, the idea is to write something like

@traits Base.map(f, a::Vector) = @old_implementation(Base.map, Tuple{typeof(f), Vector}, f, a)
# and now we could use @traits base dispatch to add detail without overwriting the base definition
@traits Base.map(f, a::Vector) where {isempty(a)} = "my_specialization" 

This has the disadvantage of being append-like in the sense that it accumulates a larger and larger function when re-executed, as the purpose is to exactly overwrite what @old_implementation graps.

I think it is better to just repeat the concrete code and spend the maintenance effort, like

@traits function Base.map(f, a::Vector)
# old implementation
end

Or define your new function in a new module

module MyModule
@traits map(f, a::Vector) = Base.map(f, a)
@traits map(f, a::Vector) where {isempty(a)} = "my_specialization" 
end # module

[tim.holy]

Done “right” you’d only need @traits Base.map(f, a::Vector) where {isempty(a)} = "my_specialization" and it would automatically take the old definition and convert it into the fallback. And do it all by dispatch, i.e., it’s no different than your hand-written versions above.

But the reason I have growing skepticism about being able to “steal” a method and make it traitable is different from anything mentioned so far: it’s because it potentially opens it up to a whole bunch of new method ambiguity/method ordering errors. For example, if your map above had been written against AbstractVector rather than Vector, and I create MyVector with a specialized version of map, for an isempty(::MyVector) which version should win? For that reason I’m thinking that before we really “do” traits we might need to be able to write (in manual-traits syntax):

foo(@prioritize(A::AbstractArray), moreargs...) = foo(Traits(A, IsStrided(A), IsContiguous(A)), moreargs...)

and essentially block someone from specializing foo(A::MyVector, moreargs...), forcing them instead to implement foo(A::Traits{<:MyVector, <:IsStrided, <:IsContiguous}, moreargs...). For such methods, traits become an official and obligate part of foo’s API, and the person extending the API has to deal with resolving ambiguities at the time the extension is designed.

This basically introduces a requirement for the community to agree on which traits, in which order, are the things we want to dispatch on. Some will justifiably say this would be a major bottleneck to development. However, I worry more about a potential future in which everyone has “trait-ized” the same method differently, and suddenly a whole bunch of our packages can’t be used in combination anymore. IMO that would be a much bigger catastrophe.

[schlichtanders]

ambiguities

@tim.holy

For example, if your map above had been written against AbstractVector rather than Vector , and I create MyVector with a specialized version of map , for an isempty(::MyVector) which version should win?

So lets say I’ve defined

module MyModule
@traits map(f, a::AbstractVector) = Base.map(f, a)
@traits map(f, a::AbstractVector) where {isempty(a)} = "my_specialization" 
end # module

and you define

@traits MyModule.map(f, a::MyVector) = "yours default"
@traits MyModule.map(f, a::MyVector) where {isempty(a)} = "yours empty"

Then this will already work out of the box without any need to solve ambiguities. The reason is that Traits.@traits splits dispatch in an outer function, doing the standard julia dispatch, and an hidden inner function, doing the extra dispatch. As the dispatch on MyVector vs AbstractVector is a standard one, it is resolved in the outer function, each referring to its own inner-function so to say.

---

I think this also solves the need for your suggestion

@tim.holy

we might need to be able to write (in manual-traits syntax):

foo(@prioritize(A::AbstractArray), moreargs...) = foo(Traits(A, IsStrided(A), IsContiguous(A)), moreargs...)

and essentially block someone from specializing foo(A::MyVector, moreargs...) , forcing them instead to implement foo(A::Traits{<:MyVector, <:IsStrided, <:IsContiguous}, moreargs...) .

A @prioritize is not needed, as there is the distinction between first (aka outer) dispatch (i.e. standard dispatch) and second (aka inner) dispatch (i.e. extended traits dispatch).

default fallback

@tim.holy

Done “right” you’d only need @traits Base.map(f, a::Vector) where {isempty(a)} = "my_specialization" and it would automatically take the old definition and convert it into the fallback. And do it all by dispatch, i.e., it’s no different than your hand-written versions above.

looks like this is already kind of possible with your own code GitHub - timholy/CodeTracking.jl: It's editing-time, do you know where your methods are?, concretely I mean the definition() function.

Envisioned steps to actually support this:

1. if a new traits definition is defined, check whether there already is a traits-fallback (there is one fallback per standard signature, i.e. one fallback per possible standard dispatch)
2. if a fallback is already defined, stop here. Else check whether the original function is already defined for the same signature using m = CodeTracking.@which ...
3. if no implementation could be found, stop here. Else call d = CodeTracking.definition(m) respectively to extract the Expr object which represents the existing implementation.
4. take this definition and construct the default fallback, something along @traits f(...) = begin $d end

Looks great to me! I’ll try to implement that.

[tim.holy]

I’m more concerned about someone writing

MyModule.foo(f, a::MyVector) = ...

without the @traits. By default this will “win” because AbstractVector is less specific than MyVector. No technical ambiguities here, but there’s a kind of ambiguity in the sense that maybe, as the author of foo, I was really counting on controlling the dispatch path of any empty AbstractVector regardless of its type. If I can force prioritization of the trait-depot then I can essentially force people to become aware of the myriad subtleties of extending foo.

I recognize that this mixes together a couple of things, but with traits adding orthogonal axes to dispatch I’m a bit leery of making it too easy until we’ve thought a bit more about controlling ambiguities.

looks like this is already kind of possible with your own code https://github.com/timhol/CodeTracking.jl, concretely I mean the definition() function.

Looks great to me! I’ll try to implement that.

I think you’ll break package precompilation. (If two packages extend foo in different ways then the order in which you load the packages becomes important.)

[tim.holy]

To illustrate the ambiguity problem more concretely, let’s take the example of two orthogonal axes from the Traitor README. Let’s say I’m a “traits-naive” author, and in my package I write

foo(x) = 1

Everyone just loves this awesome new foo method, but they need to specialize it. Person A does the following:

# Steal `foo` and create a traits version
expr = steal_method_body(which(foo, Tuple{Any}))  # grabs the current method body then deletes the method
foo(x) = foo(Traits(x, Size(x)))
foo(x::Traits{<:Any, Big}) = 2   # a version highly specialized for Big things
@eval foo((x::Traits{<:Any, <:Size}) = $expr

But person B does this:

# Steal `foo` and create a traits version
expr = steal_method_body(which(foo, Tuple{Any}))  # grabs the current method body then deletes the method
foo(x) = foo(Traits(x, Odor(x)))
foo(x::Traits{<:Any, Smelly}) = 3   # a version highly specialized for Smelly things
@eval foo((x::Traits{<:Any, <:Odor}) = $expr

Now boom Person A’s package is completely broken just by loading Person B’s package. Ah, but wait you say, maybe we could be smart and generate the depot to do this:

foo(x) = foo(Traits(x, Size(x), Odor(x)))

And to be helpful I’ll also go through and redefine all their trait-dispatch methods to account for the pair of traits. But you’ve still broken both packages, because now you’ll get situations where you end up needing to dispatch on Traits(<:Any, Big, Smelly} and no one has prepared for this possibility or even knew it could happen.

In contrast, if the trait depot is defined from the beginning then at least everyone knows what might happen. We agree as a community on which traits we need and then everyone has to dispatch on them. Thanks to version controlled Pkg and CompatHelper updating with a new trait-axis is not the nightmare scenario it would otherwise be.

This is a very real-world issue. In the bad old days, getindex used dispatch to implement all sorts of fancy indexing behaviors (vector indexing, trailing 1s, etc.) To avoid package conflicts we had to work hard to make sure that the default getindex method was the most generic (least-specialized) version in existence, and also give people a way to write just the specialization they needed. This is where Vararg{Int,N} came from, so you can write Base.getindex(A::MyArrayType{T,N}, i::Vararg{Int,N}). That’s basically the foundation of Julia’s array-awesomeness, and the most difficult problem was figuring out how we were going to support fancy generic behaviors while also allowing people to specialize for their specific array types. What we needed to do was engineer an absence of methods (by requiring high specificity from the methods that would be implemented) so that generic fallbacks with low precedence nevertheless end up being called.

[schlichtanders]

@tim.holy

But you’ve still broken both packages, because now you’ll get situations where you end up needing to dispatch on Traits(<:Any, Big, Smelly} and no one has prepared for this possibility or even knew it could happen.

I fully agree that you have to define this extra method for things which are BOTH Big and Smelly if you overloaded the same function with a specialization for each Big and Smelly separately. (This is also true for the current implementation of Traits.@traits)

But I do not agree that this is by surprise, or that no one was prepared for this. It is standard Julia dispatch disambiguation. The same hits you when you work with containers and suddenly your concrete type definitions are not enough any longer, but you need to add the abstract dispatch to solve ambiguation problems (like, say Int, Float32,… And then you also need Number).

It actually surprises me, how surprising you find this

This only happens if some user actually wants to combine the Smelly and the Big world, so for sure that user also best knows what actually should happen if a thing is both Smelly and Big. He/she is the best one to decide how the definition of this interaction looks like.

[tim.holy]

It actually surprises me, how surprising you find this

I’m not surprised by it; the issue is that I worry it will take a fairly rare problem and makes it a common one. For arrays, I can easily imagine dispatching on at least IsStrided, IsContiguous, and StorageOrder, before even thinking about traits that are connected to the eltype. (We might want PrecisionBits to generalize from Float64 to Dual{Float64}, IsScalar, SIMDFriendly, etc.) Anyone who has defined a new matrix type and then tried writing linear algebra methods knows that one suddenly has to handle Adjoint{<:MyMatrixType}, what to do for *(::Diagonal, ::Adjoint{<:MyMatrixType}), etc. That this is necessary makes perfect sense, but nevertheless I suspect that few of us would say that handling all those methods is our favorite part of Julia. Consequently, the problem I’m intrigued by is how to get the good stuff without the bad; what new stuff do we need in order to avoid a combinatorial explosion of methods whose only purpose is to resolve ambiguities? That, I think, is the really interesting problem in this space.

To clarify, I don’t think it’s unsolvable: like you say, we’ve been dealing with this anyway (https://docs.julialang.org/en/latest/manual/methods/#man-method-design-ambiguities-1). But as one example of what I think might be useful: we might want a TraitCascade type that looks something like

struct TraitCascade{X,N,Trts,Handled}

Handled would actually be a tuple like (true,false,false), indicating that we’ve finished dispatching on the first trait-axis and are ready to make our next decision based on the second trait-axis. This would be like an automated version of https://docs.julialang.org/en/latest/manual/methods/#Dispatch-on-one-argument-at-a-time-1. Sometimes you really need to engineer the absence of methods that will cause ambiguities (in this case, a community expectation that no one will ever dispatch on the first trait when Handled[1] is true), and tricks like these can help a lot.

[schlichtanders]

thanks a lot for all your patience explaining the hidden problem you see!

For the problems I worked with so far, I indeed had only a rather small space of traits I wanted to combine. But for things like Arrays the picture is massively different, as I now understood. There, many different users / packages want to experiment with custom traits, leading to a potentially very large amount of different traits.

[schlichtanders]

I added an extra section to the README, describing how you can dispatch on whether functions are defined for certain types or not.

Please see the section dispatch-on-whether-functions-are-defined—using-isdefjl

[Aremk]

There have been some discussions around inferring hasmethod by @oxinabox and @NHDaly here
https://github.com/JuliaLang/julia/pull/32732

also @bramtayl was interested in this

[ExpandingMan]

This is looking pretty cool. Any plans to add this to tag and add to the general registry? Is there a name conflict perhaps?

[oxinabox]

My thing that allows static dispatch on has_method
is now in a package

oxinabox/Tricks.jl/blob/master/src/Tricks.jl

AFAICT it works

[schlichtanders]

thanks @ExpandingMan for asking.

As far as I know there is no current name-conflict in the General Registry.
I only need to update a couple of dependencies and then port everything, one after another to the General Registry. It is going to happen somewhen soon.