Macro to "steal" types from another package

Package A defines a bunch of types whose interface I would like to extend in Package B. However, to do this in a straightforward way, Package B requires those types to subtype B.SomeAbstractType. To mitigate this I am planning to automatically generate a bunch of new types in Package B - one for each type in Package A, which wraps each of the types in Package A and passes through their properties (that is, getproperty(object_in_B::TypeInB, name) = getproperty(getfield(object_in_B, :atomic_object), name). Similarly, I will “forward” the methods in Package A that I need access to in Package B.

I’m supposing macros already exist to do this kind of thing and I wanting someone to point me in the right direction.

Extra challenge. In my use case, I actually want the wrapper types to be mutable but the atomic types are immutatable. So a bonus, if that can also be taken care of.

2 Likes

Since nobody answered in 16h, I’ll give my 2c…

This part may be easier to address by defining a union type in B that lists all the types you are interested in A, instead of trying to put them under the umbrella of a new abstract type:

julia> module A
           export A1, A2
           struct A1 end
           struct A2 end
       end
Main.A

julia> module B
           using ..A
           ATypes = Union{A1,A2}
           f(x::ATypes) = 1
           g(x::ATypes) = 2
       end
Main.B

But, probably this does not solve all your issues with dispatch. But anyway, that’s an idea. In the worst case it will bump your question :slight_smile:

1 Like

@lmiq Thanks for chiming in. Yeah, that’s not really what I want.

@ablaom I guess you are talking about the TableTransforms.jl types as the immutable types and the wrapper types in MLJ.jl as the mutable types as discussed in Hyperparameter tuning · Issue #67 · JuliaML/TableTransforms.jl · GitHub?

Why not work together in TableTransforms.jl to make these features more self-contained instead of introducing a “hacky” solution that involves stealing these types?

At the end of the day I understood that all you need is a mechanism to retrieve parameters and update them in structs in a hyperparameter tuning context?

We can certainly include hyperparameter tuning features in ways that are agnostic to specific ML frameworks.

cc: @eliascarv

1 Like

Yes, @juliohm that is my use case. The problem is that currently models in MLJ must subtype MLJModel (and transformers subtype either Unsupervised or Static). I would like to change this requirement (abandon type heirarchy in MLJ in favour of pure traits) but I estimate a considerable disruption and some of my colleagues are not enthusiastic about this. So I don’t this happening soon. (By the way, it looks to me like transformers in TableTransforms.jl also have a local abstract type. Is subtyping this a hard requirement? Perhaps you can say more about that?)

One alternative would be for the TableTransform objects to subtype from the current MLJModelInteface.jl abstract types. I assumed you (quite reasonably) were trying to avoid that.

The solution I have in mind was used to “steal” LossFunction.jl types into MLJ. It seems to have worked out in that case but I agree this not ideal. Perhaps we should just keep integration on the backburner until both MLJ and TableTransforms mature further?

That is precisely it @ablaom, I think we came across this issue before in the MLJ.jl issue tracker. There we discussed how difficult it is to work generically with models using a type hierarchy when the language doesn’t support multiple inheritance. I also wish we had a trait-based ML framework where we could mark a single model with multiple traits instead of forcing a single parent type. Hopefully, this will be sorted out in the future, even if it is a breaking change. This is a serious limitation of the MLJModelInterface.jl that could be brainstormed and addressed in a separate experimental package.

In TableTransforms.jl, we introduced a set of abstract types in order to conveniently write fallback methods (e.g. Transform, Stateless, Colwise). They are not necessary in the sense that 3rd-party packages only need to implement the exposed API (a set of functions) for their own transform types (not tested). In other projects, however, I could not escape from a parent type in Julia because the fallback code was quite complex to reimplement (e.g. Meshes.jl).

Yes, like with LossFunctions.jl, we are trying to write a self-contained package for tabular transforms that is agnostic to specific frameworks. I believe that there is great value in implementing generic modules in Julia that compose well instead of trying to solve everything in an umbrella package. For instance, LossFunctions.jl and TableTransforms.jl are already being used in GeoStats.jl, which is an umbrella package for “Geoscientific ML”. I could have implemented these features inside our umbrella instead, but that would inhibit other umbrellas from benefiting from the work. Also, keep in mind that first-time users can contribute much more easily to self-contained packages as opposed to huge umbrella projects. Given our community size, this is super relevant.

I am happy that you are interested in these efforts, and so feel free to proceed with the planned integration if you feel that is the best use of your time right now. My impression is that our community is so small that the time you will spend trying to “wrap” these modules in MLJ-specific types could instead be used to polish a really nice hyperparameter tuning interface for the TableTransforms.jl types that is agnostic to MLJ.jl, GeoStats.jl or any umbrella project, and yet meets all the requirements of real-world applications.

Thanks @juliohm for the detailed and thoughtful responses. I will take your advice and not attempt an interim but inferior integration at this time.