Composition and inheritance: the Julian way

General Usage
,
62

The key point here is the word just. For a simple example you’re definitely right. But for a real life application (such as wrapping a DataFrame object as we discussd above) this implies redefining ~200 methods. Of course a few of these (namely all those returning a DataFrame struct) have to be redefined anyway, but what remains is still quite a large number.

Of course this is feasible, either manually or through macros, but my first thought is that I should try to find an alternative method. This triggered all present discussion.

Yeah, clearly all what we say here applies only if a structure is involved.

Of course I don’t want to fight any train. I’m asking here because I recognize many here are as way more expert than me. Still: no one told me yes, you have to blindly forward those ~200 methods, and forget about inheritance.

Which is OK for me, but I didn’t saw this carved in stones, nor in the manual or in the discourse or anywhere else. Hence my doubt arose…

To answer @favba: in your proposal you are implementing the name, age and set_age for both Person and Citizen, which is exactly what I wanted to avoid (the famous ~200 methods…). Regardless of macro usage: the methods are always there: they do nothing, they flood the dispatch table and they have to be compiled.

2 Likes
63

Thanks @ScottPJones!!!
You’re solution is definitely a very interesting one, and I think it definitely solve the problem in the first discussion of this post. You’re great!! :+1::+1:

In brief, your solution uses composition of structures and avoid useless methods re-definition.
Clearly, as noted by @ChrisRackauckas, there are quite a lot of getproperty calls involved. However (correct me if I’m wrong) there are quite a lot of getproperty calls involved also in method forwarding isn’t it?

64

One apparently officially questionable way is discussed in https://discourse.julialang.org/t/getproperty-decorations-inheritance-in-0-7/11237.

65

No, you don’t have to. If your interface on the abstract type is done via accessor functions (or lazy properties), then the interface to implement is tiny and limited to defining those accessor functions. See something like the AbstractArray interface which takes

https://docs.julialang.org/en/v0.6.2/manual/interfaces/#man-interface-array-1

and then works in literally any function which takes arrays. This is because if you develop code for the abstract type based on the interface (and it doesn’t have to be an abstract type, for example the Iteration Interface is for anything which acts like an iterable) then any implementation of that interface will do. If all of Julia’s arrays can work with 5 methods plus some extra traits, then that shows there’s quite a lot you can do.

If your interface is 200+ methods, it doesn’t matter if it’s Julia or OOP, you’re doing it wrong!

Dataframes is a bad example because they never defined an interface for a Dataframe which is why it’s so hard to extend. If you want a table interface, look at DataStreams.jl or IterableTables.jl. These have only a handful of functions and any source/sink will then work as a table. For example, Dataframes implements the source while StatPlots.jl implements the sink, so those two packages work together. But DifferentialEquations.jl solutions are also a sink, so you can convert its solutions to Dataframes or use it just like a table due to the 3 methods implemented by David. So, 200 methods >> the 3 that are used in the real live actually working example :smile:.

Here’s the full code that makes every differential equation solution (ODE/SDE/DAE/DDE/jump/etc.) have a table interface and directly convert to DataFrames.

https://github.com/JuliaDiffEq/DiffEqBase.jl/blob/master/src/tabletraits.jl

1 Like
66

Now the bad news: despite the very clever suggestion by @ScottPJones this solution can not be applied in general, since it works only if the object you wish to extend/customize already uses the person(a::Person) = a method internally, which doesn’t apply to DataFrame…

1 Like
67

It seems that there are two tasks, then for DataFrames, if you were to create a new type for metadata (which you don’t have to do! maintainers seem amenable to metadata living in DataFrames)

First, make sure DataFrames methods only take in AbstractDataFrame. Second, categorize which functions absolutely need to be defined toe AbstractDataFrame to work. Maybe there already exists a list somewhere? Or we could work on getting that list together and shortened. A good start would be to create a new MyDataFrame <: AbstractDataFrame type that is a mirror image of a DataFrame type and seeing how that works.

68
69

…and this provides a motivation for the second part of the question. thank you for clarifying! :+1:

70

Good to know. Actually, there already is a PR for this, although it has been assigned no reviewer yet…

Yeah, I can try to do it if some of the maintainers manifest their wish to go in that direction, so that there is some chance that I will not waste my time in a PR which will surely be rejected.

71

So it appears we’re finally figuring out a solution to the problem here. I will try to summarize:

  • the problem arose trying to extend/customize the behaviour of a Julia object (i.e. a structure+methods, call it Person) into another one (call it Citizen) in a way that it can be used exactly as the base type, except for a few twist (i.e. specialized methods);

  • we wanted to solve this problem minimizing as much as possible the number of methods redefined just to forward calls to the base type;

  • minimizing this number implies we had to implement an inheritance-like approach to the problem, and we identified a few rules to be followed (here and here);

  • the inheritance-like approach, however, turned out to be non convenitent as discussed by @ChrisRackauckas here and here. The best, and more Julian way to face this problem is by means of composition, regardless of the number of method to be re-defined;

  • @ScottPJones found what I think is a very nice approach here which uses both composition (in place of inheritance) and minimizes the number of method re-definitions. However, this method requires to re-write the methods in the base class;

  • The real interest for this discusion was the possibility to customize a DataFrame object (further discussion here) and refactor the whole package following @ScottPJones solution is not feasible;

  • @ChrisRackauckas provided the last piece of information: the DataFrame still lacks the definition of an interface which simplifies extending it. Here’s why we had all the problems which ultimately triggered this discussion.

The whole story was absolutely not clear to me when I first started this post, and I’m sorry if everything was obvious to many people here. I learned a lot and wish to thank all of you! :+1

14 Likes
72

I am curious if @ScottPJones’s method could be formalized. By methodology or by language (macros) support.

I am also curious if it will make negative performance impact comparing to “slavery” work with ~200 methods rewrite.

Scott’s method allow multiple inheritance too! (So it could also bring many problems which multiple inheritance use to bring)

1 Like
73

Dear all,
this topic has been inactive for a few months, but I just wanted to add that I implemented a package to solve the first problem posted, namely how to implement some form of inheritance mechanism in Julia.

Actually, the solution is to implement subtyping which, although not equivalent to inheritance (e.g. here), allows to quickly and elegantly solve the problem.

The trick is to unambiguously link an abstract type and a concrete one, realizing a kind of quasi-abstract type. Then we can use the latter as argument type for methods: it is both an abstract type (allowing the same method to be called on subtypes) and a concrete one (ensuring the method will receive the appropriate concrete structure). A macro can then takes care of copy/pasting the fields from one quasi-abstract* type to another, realizing a concrete subtyping pattern.

The details are discussed in the README.md file of the ReusePatterns package.

The implementation of concrete subtyping for the first example in this post is as follows:

using ReusePatterns

#---------------------------------------------------------------------
# BASE TYPE : Person
@quasiabstract mutable struct Person
    name::String
    age::Int
end
Person(name) = Person(name, 0)
Base.display(p::Person) = println("Person: ", p.name, " (age: ", p.age, ")")

function happybirthday(p::Person)
    p.age += 1
    println(p.name, " is now ", p.age, " year(s) old")
end

call(p::Person) = print(uppercase(p.name), "!")

son(p::Person) = Person(p.name * " junior", 0)

# EXAMPLES
p = Person("Giorgio")
happybirthday(p)
call(p)
call(son(p))


#---------------------------------------------------------------------
# DERIVED TYPE : Citizen
@quasiabstract mutable struct Citizen <: Person
    nationality::String # new field
end

# Note that the fields associated to `Citizen` automatically include those from `Person`
println(fieldnames(concretetype(Citizen)))

function call(p::Citizen)
    if p.nationality == "Italian"
        print(uppercase(p.name), " dove sei ?")
    elseif p.nationality == "UK"
        print(uppercase(p.name), " where are you ?")
    else
        invoke(call, Tuple{supertype(Citizen)}, p)
    end
end

Citizen(name, nationality) = Citizen(name, 0, nationality)

function son(p::Citizen)
    person = Person(getfield.(Ref(p), fieldnames(concretetype(Person)))...)
    return Citizen(person.name * " junior", 1, p.nationality)
end

p = Citizen("Giorgio", "Italian")
happybirthday(p)
call(p)
call(son(p))

p = Citizen("Kim", "UK")
happybirthday(p)
call(p)
call(son(p))

p = Citizen("Jean", "French")
happybirthday(p)
call(p)
call(son(p))
3 Likes
74

There are also the packages ConcreteAbstractions.jl and Classes.jl that do similar things.

1 Like
75

Yes, I cited both packages in the links section of ReusePatterns, in order to encourage the users to check for similar functionalities.

However, ConcreteAbstraction.jl seems to be unmaintained.
Class.jl is alive, and the main difference is that it operates in several points in the user code (once in the structure definition and once for each method) while ReusePatterns provides just one macro (@quasiabstract) to be used in the structure definition. Moreover, ReusePatterns aims also to provide a direct comparison with the composition pattern, to check which one provides the best solution.

Finally, it provides a (hopefully) useful discussion on the rationale behind the composition and concrete subtyping approaches. It is surely due to my ignorance, but when I first started this post I was not aware of the many subtleties involved, and if I had the opportunity to read something like the README.md which is now available in ReusePatterns I would not have started this post… :wink:

3 Likes
76

By a quick count in this thread, there are 5+ packages to help solve this problem. Curious if there is anything in the Julia roadmap or comments by the developers on addressing this in the language ( or Base/Standard Library)? While packages are nice, it seems there is a strong case for it to be addressed in the core language ( even if it’s just in the documentation ).

As a bit of an aside, I find the concept of traits extremely useful in arriving at an optimal Julian solution. I often start implementing trait-based dispatch only to find a refactor of my types/functions that does the same thing without traits.

2 Likes
77

A few questions!

What’s the connection between @forward and @quasiabstract? I may misunderstand but it seems like this would also make sense as two packages, one for composition and one for concrete subtyping? I’ve always thought there should be a Forward package that just does @forward macros well.

I also use Mixers.jl for field/type/macro inheritance and traits take care of method inheritance for the mixin. What would I gain from switching to quasiabstract inheritance? The looseness of using mixins does bother me occasionally but it has the advantage of being very simple and applicable on unrelated branches of an abstract type hierarchy.

Edit: Also how does concrete subtyping play with Parameters.jl or FieldMetadata.jl style macros where struct fields are annotated?

78

Another recent package for inheritance that is not mentioned in this thread. Is one I have been working on StructuralInheritance.jl.

It works by creating a concrete type and an abstract type which is used to inherit behavior. It stores the fields that are defined with fully qualified types and splices it into inheriting types.

julia> using StructuralInheritance

julia> import StructuralInheritance: totuple #used for immitating superconstructors

julia> @protostruct struct A
           firstFieldFromA::Int
           second::Float64
       end
ProtoA

julia> @protostruct struct B <: A
           yetAnotherFieldName::Float64
       end "SomeOtherPrefix"
SomeOtherPrefixB

julia> @doc B
  No documentation found.

  Summary
  ≡≡≡≡≡≡≡≡≡

  struct B <: SomeOtherPrefixB

  Fields
  ≡≡≡≡≡≡≡≡

  firstFieldFromA     :: Int64
  second              :: Float64
  yetAnotherFieldName :: Float64

  Supertype Hierarchy
  ≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡

  B <: SomeOtherPrefixB <: ProtoA <: Any

super constructors can also be imitated

julia> A(x) = A(x^5,x^(-1.0))
A

julia> B(x) = B(totuple(A(x))...,x)
B

Functions defined on the abstract type that is created with A, will work with anything that uses @protostruct to inherit from A or any of its descendants.

julia> dosomething(x::ProtoA) = x.firstFieldFromA * x.second
dosomething (generic function with 1 method)

julia> B(2)
B(32, 0.5, 2.0)

julia> dosomething(B(2))
16.0
79

I agree: we need to formalize the way we approach the problem, and document it once and for all. As I said above, the most useful part of ReusePatterns.jl is probably its README.md file…

2 Likes
80

Both @forward and @quasiabstract allow to implement reusing patterns. But what is the best pattern depends on the problem and the boundary conditions (i.e. if Alice and Bob talks to each other…).

Hence it makes sense to provide them in a single package to facilitate switch back and forth, and to provide a clear comparison of the approaches in the examples.

AFAIK, the @forward macro in ReusePatterns.jl is the only one which automatically forwads all the necessary methods.

I’m not sure you will gain anything operationally or in terms of performance. As I said several similar functionalities are provided by other packages.

The point is: do you simply want your code to work, or do you wish to follow a (hopefully) well thought approach (even if it doesn’t provide any practical advantage) ? In the former case you may choose any package you like. In the latter case I suggest you to give Reusepatterns.jl a try, and if you find some flaw in the reasoning please tell me. I’m way more interested in understanding how it should be done, rather than in the implementation itself.

1 Like
81

I just gave it a try, and unfortunately it doesn’t work.

As far as I understand both @with_kw and @metadata are just convenience macros, although complex. On the other hand the code generated by @quasiabstract is very simple (see the examples), hence if you should use them all toghether I would simply avoid using the quasiabstract macro, and do typing by hand.

As I stated above, the main purpose of ReusePatterns is to propose standard patterns for code reuse, not necessary implemented with ReusePatterns.

If concrete subtyping should gain popularity I may consider the idea of making ReusePattern compatible with other widely used packages. As always PR are more than welcome… :wink: