Trojan horse types - a type that takes over the function calling it

Internals & Design
#1

I have been implementing a new type of array which is just a normal array except the content is fetched in the background and everything about it is the same as a normal array including access etc.

I call this Array the Ghost Array. For example I have been defining functions like

getindex(a::GhostArray, I) = getindex(materialize(a), I)

So I am overloading the getindex method which is fine. But I want GhostArray to work with hundreds of methods and the pattern for doing so is the same which is.

fn(a::GhostArray,kwargs...)=fn(materialize(a),kwargs...)

So I want for all fn, do the above. But there is no way to know all possible fn. And people can always define more fns. Also there are fns where GhostArray is not the first argument. I want to “overtake” them too.

Solutuon
One way I thought would be to have some Trojan horse types that can take over the any function and replace itself with materialize(a).

I call these types the Trojan horse types, because they go in and try to take over.

I don’t think this can be done in Julia yet. I know of Cassette.jl but I think that would require the code to be run in a context so can’t deal with all possible functions (my understanding might be off here).

I would highly appreciate if anyone can point to programming language research that already has this idea discussed and researched this and if there are languages already implementing this.

Would be nice if something like this is possible in Julia though. Or maybe it is already! Please enlighten me!

#2

New thought:would this be an area where tradition OOP can deal with? Just make GhostArray inherit from Array?

#3

Can you make some pseudocode of what you want it to do? Maybe a few dumby examples. I think I know what you want, but don’t want to completely miss the point with my answer.

1 Like
#4

The AbstractArray interface isn’t actually that large. If you have defined a few key-methods, most stuff will work through generic fallbacks.

3 Likes
#5

Is there a listing of the interface like with the Iterator interface?

I think implementing the interface are fine, but I think my code should work on any function.

#6

Do you have an example where your code fails?

#7

Well I have a type called TypeA <: SomeAbstractType, now I want to

For every fn(..., a::AbstractType, ...), I want to overload it so that

fn(..., a::TypeA, ...) = fn(..., materialize(a),...)

#8

Good point. Mostly it fails, becaue I haven’t implemented an interface function.

But the concept is more general and should apply to any type, not just Arrays with well defined interfaces.

#9

i don’t know if i have the same problem, but is at least similar.
I have a function $property(model,volume,temperature,mol_fraction) where mol_fraction is an array. i have a subtype of AbstractArray called AbstractMaterialVector (takes care of conversions between mol and mass quantities) and i need to do something like

$property(model,volume,temperature,x<:AbstractMaterialVector) = $property(model,volume,temperature,to_mol_fraction(x))

for now, i use an @eval loop with every property function that i have, but i need to update that constantly

1 Like
#10

You can get a hint of what you need to implement by calling methodswith(Array)

1 Like
#11

Interfaces, proper use of Abstract type trees and promotion should get you quite far. Once this is no longer the case, loops with @eval is as far as I know the goto solution. If there is a better way I would certainly like to know, as I make extensive use of this strategy

4 Likes
#12

I think that’s a good practical solution. I think I am getting is that it’s not possible in Julia to this in general.

You can do this for all functions that exists already. Possibly using a macro, but there is no way to do this for all functions. Unless you put it inside context and use Cassette.jl

Which is fine. Just confirming that it’s not possible is good enough for me

#13

That’s the issue isn’t it. A trojan-horse type concept will solve this. Or some sort of inheritance-based magic.

1 Like
#14

Could you overload splat, with a ternary operator? Caveat, all fn args would need to be supplied as tuples. Maybe splat is one of those protected thingy madoodles.

If I understand it right, you want a different behavior for a base type at the value level?

1 Like
#15

Thanks for the pointer to splat, was unaware of this little gem :gem:

#16

Why not

abstract type GhostFunction <: Function end
(fn::GhostFunction)(a::GhostArray, args...) = fn(materialize(a), args...)

struct SomeFunction <: GhostFunction end
const somefunction = SomeFunction()
somefunction(a::Array, args...) = ...

? (I wish the last lines can be done with function somefunction <: GhostFunction end or something.)

Though this cannot be used with existing functions like getindex.

Related?

Spoiler: @​JeffBezanson has a convincing argument against this direction so I guess it is not likely to happen.

1 Like
#17

Very cute approach. But it’s the same? I have to overwrite every function with GhostFunction also, what if a doesn’t appear in the first position?

#18

If there are n definitions like fn(a::GhostArray, kwargs...) = fn(materialize(a), kwargs...) for m functions like fn, GhostFunction-based approach requires O(n) + O(m) lines; you need O(nm) lines in a naive approach.

You can use map over args and materialize it if the argument isa GhostArray. Julia is clever enough to compile it away.

#19

This question comes up very frequently. I think that the right solution to this problem is to define and document an interface that a particular thing has to comply with, and keep it small.

If you just want to implement an <:AbstractArray, I am not sure where those “hundreds ot methods” are coming from. Ideally, if they work for an <: AbstractArray, and you implement the API for the latter, they should work for your type, too.

5 Likes
#20

One option is the use of a funxtion thst dispatches on that argument:

getvalue(a) =a
getvalue(a::GhostArray) = materialize(a)

And if the defined methods are yours, then you just use getvalue everywhere

1 Like