"Best practice" type piracy for package compatibility?

General Usage
,
1

I would love to know your opinion, whether the (pretty safe) type-piracy as described below is in your opinion OK, and whether this mechanism via Base.convert() could become a “best-practice” example to solve the issue of glueing two packages together? Maybe it can even be supported by some convenience macros in the future?

Often a package FunctionProvider provides a certain functionality (e.g. a function) that are potentially useful if combined with another package MyStructProvider that hosts a particular datatype. However, the makers of FunctionProvider may not be aware of MyStructProvider and MyStructProvider may not be interested in importing FunctionProvider, since this may be a heavy package. The common compromise is for FunctionProvider to also provide a second lightweight package, which is then imported by MyStructProvider to allow writing the necessary glue code (e.g. rewrap the type) to make functions of FunctionProvider work with the datatype of MyStructProvider.
However, even this lightweight package needs maintenance and MyStructProvider may not want to include even this lightweight package since only few users may need this particular package combination.

Therefor I looked for a mechanism with a little bit of (fairly safe) type piracy that may provide a solution:

FunctionProvider.jl:

module FunctionProvider
export foo
function foo(something::Int)
    print(something)
end
# here comes the code allowing "legalized type piracy".
TransferType = NamedTuple{(:FunctionProvider, :transfer), Tuple{Nothing, NamedTuple{(:data,), Tuple{Int64}}}}
function foo(something)
    something = convert(TransferType,something)
    foo(something.transfer.data)
end
end

Note that FunctionProvider.jl does NOT know about MyPackageProvider
MyPackageProvider.jl:

module MyStructProvider # does NOT import FunctionProvider and does not need FunctionProvider in its Dependencies
export StructProviderType
export convert
struct StructProviderType
    mydata::Float64
end

using Base
TransferType = NamedTuple{(:FunctionProvider, :transfer), Tuple{Nothing, NamedTuple{(:data,), Tuple{Int64}}}}
function Base.convert(::Type{TransferType}, something::StructProviderType)::TransferType 
    converted = round(Int, something.mydata)
    return (FunctionProvider=nothing, transfer=(data=converted,))
end
end

And here is an example, which demonstrates the use of this mechanism:

push!(LOAD_PATH, pwd())  # just to avoid adding the packages to the environment
using FunctionProvider
using MyStructProvider

a = StructProviderType(22.2)
foo(a)

Please don’t forget to change directory into the bare directory, where these 3 files are located.
Note that this glue, i.e. defining the Base.convert() function, can also be provided from outside both of these packages. What is the actual use-case? It would be nice, if my View5D.jl package provides an easy-to-implement mechanism that other datatype packages can include their meta information on units, axenames, pixelsize, offset etc. and choose default values (e.g. gamma) for direct visualization via the View5D.jl macros e.g. @vv myLocatedArray.

Much looking forward to your opinion. Why do I call it “pretty safe”? Well, the type that is converted to has the destination package name “FunctionProvider” in its type (without requiring an import) and both FunctionProvider and MyStructProvider know about this use.

2

Can I suggest “type privateering”?

Imo this is a pretty obscure pattern. Having worked on a few glue packages myself I understand the impulse, but it’s going to be real difficult for someone who reads the code to understand what it does.

I’m just holding out for https://github.com/JuliaLang/Pkg.jl/issues/1285

2 Likes
3

Great Suggestion! I like „type privateering“ as it captures the intention of this code: Provide a reasonably safe mechanism for other packages to write a few lines of glue code to achieve compatibility of their type to the type of a given package. All of this is achieved by a mutually agreed „privateer type“ being a named tuple with the first entry being the name of the package owning the type to convert to.

I guess from a user perspective this can be fairly easy as one could provide simple instructions about how to write such conversion code.
The cool thing: Not a single import needed.

I wonder how this mechanism compares to the other mechanisms in terms of precompilation etc…

4

I think it is a very interesting pattern. You use a function and a parametric type from Base to create an interface between a Data and an Algorithm packages without any need to include the other. The Algorithm package needs to subscribe to the pattern, and the Data package needs to be aware both of the pattern and of the Algorithm package. I like that both Base.convert and Base.NamedTuple does not seem to have been perverted from their semantic purpose.

The only “problem” that I see, is that if the Algorithm package does not make an effort to provide a lightweight names package then it is possible it does not make an effort to subscribe to this pattern too.

Also, this is more limited than pirating in the sense that you cannot extend the Algorithm to a completely different behavior with a new type, instead you convert the new type to a type Algorithm already knows how to deal with, right? Obviously, the Algorithm can take an abstract type so, your conversion could convert your new type to a new wrapper type that inherits that abstract type. I think this would be specially nice because it works around the “problem” of single inheritance in Julia.

However, if the abstract type is defined at the Algorithm package (instead of Base, for example, Number) then we go back to the original problem, that the Algorithm package should provide a lightweight package with the types it uses (even if they are just abstract) and that the Data package should import it to be able to make types that inherit those.

But then it would be “real” pirating again, no? Because none the type nor the function are “owned” by the third package, and you are creating a spooky-action-at-distance (i.e., the third package changes the behavior of code that is both (i) unaware of the third package but (ii) aware of the other two packages). What happens if two third-party packages decide to provide this glue code for their purposes? I think the last to be loaded wins (i.e., overrides the previous glue).

I think the pattern works around the lightweight package problem (partially) and may have some interesting uses working around single inheritance, but it does not seem to me to solve the third-party glue problem.

5

Another issue with this approach is that package dependency provides synchronization. What if the algorithm package decides that now it needs another privateering signature. The data package would need to update, but there are no compat entries to tell a user which versions of the packages work together.

1 Like
6

Providing such a package is not so much the problem and you are completely right, the algorithm provider will then also not make the effort for this scheme of subscribing. However, I was more worried about the user side. Specifically, View5D visualises up to 5D arrays, also displaying meta information such as pixel location and brightness units etc. There are probably tons of packages, which could benefit from this kind of visualization, even if just for debugging reasons. Yet, few may want to “drag along” even a light-weight package but if I send them a nudge “would you mind putting the following into your code somewhere” to allow your code to be easily displayed, maybe they will agree?
Of course I could implement all the various glue codes in View5D but this would then drag all the various packages along, which is clearly inacceptable.

7

Yes, this is true. For my use-case visualization during debugging, this would probably not be a problem as long as the code in the end performs the correct task (display the datatype via view5d) and this is what the View5D package explicitely allows others to do, but I agree that this is a downside of the general concept.

One can think about including a name-tag of the defining package, but it would not help here as long as it is not part of the caller syntax. Maybe this can be done by some kind of introspection (find out the calling module in the function provider and dispatch to a type that contains this caller module name?
But this would probably then hinder the original use in debugging :wink:

8

Very valid point. The function provider would really need to stay downwards-compatible with its signature.
Well, it could provide a hand-coded warning or something like this. But I agree that this is not a great solution for a general system on the long run.

9

I might be saying nonsense, since I haven’t used your package and don’t know which packages which could be integrated with yours. But I would try a different approach, e.g. using Requires.jl.

For instance, you could create a generic function, say, data_for_view5d, then view_function(arg) = view_function(data_for_view5d(arg)) for each convenient view_function and create one data_for_view5d method for each external “struct provider” in a separate file which is loaded (by Requires.jl) when the corresponding package is available.

10

You are right. But this does require the “struct provider” to do “using Requires” and have an __init__() function where the “requires” macro is called, which means its as much extra dependencies as importing a lightweight glue package at least for the first package you optionally want to interface to. I guess your solution is still a cleaner one, in terms of version changes and code readablilty, but then also not really a solution either, since gustaphes comment (see above) is also not handled without problems, right?