# Trojan horse types - a type that takes over the function calling it

**URL:** <https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731>\
**Category:** Internals & Design\
**Created:** [December 2, 2019, 1:50am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731 "2019-12-02T01:50:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 1:50am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/1 "2019-12-02T01:50:00Z")

</div>

I have been implementing a new type of array which is just a normal array except the content is fetched in the background and everything about it is the same as a normal array including access etc.

I call this Array the Ghost Array. For example I have been defining functions like

`getindex(a::GhostArray, I) = getindex(materialize(a), I) `

So I am overloading the getindex method which is fine. But I want GhostArray to work with hundreds of methods and the pattern for doing so is the same which is.

`fn(a::GhostArray,kwargs...)=fn(materialize(a),kwargs...)`

So I want for all `fn`, do the above. But there is no way to know all possible `fn`. And people can always define more `fn`s. Also there are `fn`s where GhostArray is not the first argument. I want to “overtake” them too.

**Solutuon**  
One way I thought would be to have some Trojan horse types that can take over the any function and replace itself with `materialize(a)`.

I call these types the Trojan horse types, because they go in and try to take over.

I don’t think this can be done in Julia yet. I know of Cassette.jl but I think that would require the code to be run in a context so can’t deal with all possible functions (my understanding might be off here).

I would highly appreciate if anyone can point to programming language research that already has this idea discussed and researched this and if there are languages already implementing this.

Would be nice if something like this is possible in Julia though. Or maybe it is already! Please enlighten me!

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 1:56am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/2 "2019-12-02T01:56:53Z")

</div>

New thought:would this be an area where tradition OOP can deal with? Just make GhostArray inherit from Array?

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 2, 2019, 2:08am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/3 "2019-12-02T02:08:52Z")

</div>

Can you make some pseudocode of what you want it to do? Maybe a few dumby examples. I think I know what you want, but don’t want to completely miss the point with my answer.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 2, 2019, 2:12am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/4 "2019-12-02T02:12:40Z")

</div>

The AbstractArray interface isn’t actually that large. If you have defined a few key-methods, most stuff will work through generic fallbacks.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 2:20am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/5 "2019-12-02T02:20:10Z")

</div>

Is there a listing of the interface like with the Iterator interface?

I think implementing the interface are fine, but I think my code should work on any function.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 2, 2019, 2:20am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/6 "2019-12-02T02:20:53Z")

</div>

Do you have an example where your code fails?

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 2:25am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/7 "2019-12-02T02:25:56Z")

</div>

Well I have a type called `TypeA <: SomeAbstractType`, now I want to

For **every** `fn(..., a::AbstractType, ...)`, I want to overload it so that

`fn(..., a::TypeA, ...) = fn(..., materialize(a),...)`

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 2:26am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/8 "2019-12-02T02:26:40Z")

</div>

Good point. Mostly it fails, becaue I haven’t implemented an interface function.

But the concept is more general and should apply to any type, not just `Arrays` with well defined interfaces.

---

<div class="post-metadata">

**Author:** ![longemen3000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/longemen3000/32/7298_2.png) [@longemen3000](https://discourse.julialang.org/u/longemen3000)\
**Post date:** [December 2, 2019, 2:31am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/9 "2019-12-02T02:31:24Z")

</div>

i don’t know if i have the same problem, but is at least similar.  
I have a function `$property(model,volume,temperature,mol_fraction)` where `mol_fraction` is an array. i have a subtype of `AbstractArray` called `AbstractMaterialVector` (takes care of conversions between mol and mass quantities) and i need to do something like

```julia
$property(model,volume,temperature,x<:AbstractMaterialVector) = $property(model,volume,temperature,to_mol_fraction(x))

```

for now, i use an `@eval` loop with every property function that i have, but i need to update that constantly

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 2, 2019, 2:35am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/10 "2019-12-02T02:35:44Z")

</div>

You can get a hint of what you need to implement by calling `methodswith(Array)`

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 2, 2019, 2:38am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/11 "2019-12-02T02:38:51Z")

</div>

Interfaces, proper use of Abstract type trees and promotion should get you quite far. Once this is no longer the case, loops with `@eval` is as far as I know the goto solution. If there is a better way I would certainly like to know, as I make [extensive use of this strategy](https://github.com/baggepinnen/MonteCarloMeasurements.jl/blob/master/src/register_primitive.jl)

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 2:39am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/12 "2019-12-02T02:39:23Z")

</div>

I think that’s a good practical solution. I think I am getting is that it’s not possible in Julia to this in general.

You can do this for all functions that exists already. Possibly using a macro, but there is no way to do this for all functions. Unless you put it inside context and use Cassette.jl

Which is fine. Just confirming that it’s not possible is good enough for me

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 2:40am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/13 "2019-12-02T02:40:22Z")

</div>

> [@longemen3000](#):
>
> but i need to update that constantly

That’s the issue isn’t it. A trojan-horse type concept will solve this. Or some sort of inheritance-based magic.

---

<div class="post-metadata">

**Author:** ![anon92994695](https://avatars.discourse-cdn.com/v4/letter/a/ce7236/32.png) [@anon92994695](https://discourse.julialang.org/u/anon92994695)\
**Post date:** [December 2, 2019, 2:48am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/14 "2019-12-02T02:48:36Z")

</div>

Could you overload splat, with a ternary operator? Caveat, all fn args would need to be supplied as tuples. Maybe splat is one of those protected thingy madoodles.

[https://github.com/JuliaLang/julia/blob/89a51fb98485219680c7106e94cf75fd8069f3f8/base/operators.jl#L1024](https://github.com/JuliaLang/julia/blob/89a51fb98485219680c7106e94cf75fd8069f3f8/base/operators.jl#L1024)

If I understand it right, you want a different behavior for a base type at the value level?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 2, 2019, 2:57am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/15 "2019-12-02T02:57:10Z")

</div>

Thanks for the pointer to `splat`, was unaware of this little gem 💎

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 2, 2019, 3:05am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/16 "2019-12-02T03:05:57Z")

</div>

Why not

```julia
abstract type GhostFunction <: Function end
(fn::GhostFunction)(a::GhostArray, args...) = fn(materialize(a), args...)

struct SomeFunction <: GhostFunction end
const somefunction = SomeFunction()
somefunction(a::Array, args...) = ...

```

? (I wish the last lines can be done with `function somefunction <: GhostFunction end` or something.)

Though this cannot be used with existing functions like `getindex`.

> [@anon92994695](#):
>
> Could you overload splat, with a ternary operator?

Related?

> <https://github.com/JuliaLang/julia/issues/32860>
>
> I suggest a mechanism to customize splatting behavior such that code like
> 
> \`\`\`…julia
> +((x .- y).^2...) / length(x) # compute MSE
> \`\`\`
> 
> can be executed efficiently without any allocations.
> 
> The idea is inspired by discussion in #29114 by @c42f et al.
> 
> \## Idea
> 
> Like dot-call syntax, I suggest to lower splatting to a series of function calls that are overloadable. One possibility is
> 
> \`\`\`julia
> \# f(a...) is expanded to:
> apply(f, Arguments(VA(splattable(a))))
> 
> \# f(a, b..., c, d...) is expanded to:
> apply(f, Arguments(a, VA(splattable(b)), c, VA(splattable(d))))
> \`\`\`
> 
> (To avoid recursion, this lowering should happen only when the call includes splatting.)
> 
> When the dot-call syntax appears in the splatting operand, I suggest \_not\_ materialize the dot-call. That is to say, for example, \`op(f.(xs)...)\` is lowered to
> 
> \`\`\`julia
> bc = broadcasted(f, xs) # \`bc\` not materialized
> apply(op, Arguments(VA(splattable(bc))))
> \`\`\`
> 
> This let us evaluate \`op(f.(xs)...)\` without any allocation once \`reduce\`/\`foldl\` supports \`Broadcasted\` object (#31020 started tackle this).
> 
> \## Interface
> 
> The lowering above requires the following interface functions and types:
> 
> \`\`\`julia
> function apply end
> 
> splattable(x) = x
> 
> struct VA{T}
> args::T
> end
> 
> struct Arguments{T \<: Tuple}
> args::T
> Arguments(args...) = new{typeof(args)}(args)
> end
> \`\`\`
> 
> \* \`apply\` must be dispatched on the first argument type and may be dispatched on the second argument type.
> \* Per-vararg processing should be done via \`splattable\`. This is analogous to \`broadcastable\` (@yurivish suggested this in https://github.com/JuliaLang/julia/issues/29114#issuecomment-515294850).
> \* The type \`VA\` (whose name can/should be improved) must be used only for defining \`apply\`; its constructor must not be overloaded. This is for making it hard to break splatting semantics.
> \* The constructor for type \`Arguments\` must not be overloaded for the same reason.
> 
> Using current \`Core.\_apply\`, the default \`apply\` can be implemented as
> 
> \`\`\`julia
> apply(f, x) = Core.\_apply(Core.\_apply, (f,), \_default\_splattables(x.args))
> \_default\_splattables(args) = map(x -\> x isa VA ? materialize(x.args) : (x,), args)
> \`\`\`
> 
> \## Example overloads
> 
> \### Associative binary operators
> 
> Many useful operations can be expressed using splatting into associative operators
> 
> \`\`\`julia
> +(xs...) == sum(xs)
> \*(xs...) == prod(xs)
> min(xs...) == minimum(xs)
> max(xs...) == maximum(xs)
> \`\`\`
> 
> or "mapped-splatting"
> 
> \`\`\`julia
> +(xs.^2...) == norm(xs)
> +(xs .\* ys...) == dot(xs, ys)
> \`\`\`
> 
> (This is reminiscent of the "big operator" in Fortress.)
> 
> There are also various other associative binary operators in Base. Invoking \`reduce\` with splatting could be useful:
> 
> \`\`\`julia
> const AssociativeOperator = Union{
> typoef(\*),
> typoef(+),
> typoef(&),
> typoef(|),
> typoef(min),
> typoef(max),
> typoef(intersect),
> typoef(union),
> typoef(vcat),
> typoef(hcat),
> typoef(merge),
> # what else?
> }
> 
> apply(op::AssociativeOperator, args::Arguments{Tuple{\<:VA}}) =
> reduce(op, args.args\[1\].args)
> \`\`\`
> 
> For example, concatenating vectors would be efficiently done via \`vcat(vectors...)\` thanks to #27188.
> 
> It also is possible to support
> 
> \`\`\`julia
> op :: AssociativeOperator
> op(a, bs..., cs...)
> \`\`\`
> 
> such that it is computed as
> 
> \`\`\`julia
> op(op(a, reduce(op, bs)), reduce(op, cs))
> \`\`\`
> 
> This may be implemented as
> 
> \`\`\`julia
> apply(op::AssociativeOperator, args::Arguments) = mapreduce(apply1, op, args.args)
> \_apply1(op::AssociativeOperator, args::VA) = reduce(op, args.args)
> \_apply1(op, x) = x
> \`\`\`
> 
> Note that, since \`reduce\` would degrade to \`foldl\` when the input is not an array (or not \`Broadcasted\` after #31020), we can also use it to fuse filtering with reduction
> 
> \`\`\`julia
> +((x for x in xs if x \> 0)...)
> \`\`\`
> 
> \### Non-associative binary functions
> 
> Splatting is useful for non-associative binary functions:
> 
> \`\`\`julia
> const BinaryFunction = Union{
> typoef(/),
> typoef(-),
> typoef(intersect!),
> typoef(union!),
> typoef(merge!),
> typoef(append!),
> typoef(push!),
> # what else?
> }
> 
> function apply(op::BinaryFunction, args::Arguments{Tuple{Any, \<:VA}})
> @assert !(args.args\[1\] isa VA)
> return foldl(op, args.args\[2\].args; init=args.args\[1\])
> end
> \`\`\`
> 
> or more generally
> 
> \`\`\`julia
> function apply(op::BinaryFunction, args::Arguments)
> @assert !(args.args\[1\] isa VA)
> return foldl(op, flatten(x isa VA ? x.args : (x,) for x in args.args\[2:end\]); init=args.args\[1\])
> end
> \`\`\`
> 
> \### Matrix-vector multiplications
> 
> Not sure how many people need this, but \`\*(matrices..., vector)\` can be (somewhat) efficiently evaluated by defining
> 
> \`\`\`julia
> apply(::typeof(\*), args::Arguments{Tuple{\<:VA, \<:AbstractVector}}) =
> foldr(\*, args.args\[1\].args; init=args.args\[2\])
> \`\`\`
> 
> (Of course, allocation could be much more minimized if we really want this.)
> 
> Similar optimization can be done for \`∘(fs...)\`; but I'm not sure about the exact usecase.
> 
> \### Higher-order functions (\`map(f, iters..)\` etc.)
> 
> As this mechanism let any function optimize splatting, higher-order functions that may call splatting of given function can be optimized by defining their own \`apply\` specialization. For example, \`map(f, iters..)\` can be specialized as
> 
> \`\`\`julia
> function apply(::typeof(map), args::Arguments{Tuple{Any, \<:VA}})
> f = args.args\[1\]
> iters = args.args\[2\].args
> return map(splat(f), \_zipsplat(iters))
> end
> \`\`\`
> 
> where \`\_zipsplat(iters)\` behaves like \`zip(iters...)\` but its element type does not have to be a \`Tuple\`.
> 
> Defining a similar overload for \`broadcasted\` may be possible provided that the object returned by \`\_zipsplat(iters)\` is indexable. This let us nest reduction inside mapping and avoid allocation in some cases:
> 
> \`\`\`julia
> vector .= +.(eachcol(matrix)...)
> \`\`\`
> 
> Other \`map\`-like functions including \`map!\` and \`foreach\` can also implement this overload.
> 
> \### \`print\`-like functions
> 
> \`print\`, \`println\` and \`write\` can be invoked with varargs. We can make, e.g., \`println(xs...)\` more compiler friendly and efficient when \`xs\` is a generic iterator. Note that \`apply(string, Arguments(xs))\` can also be implemented in terms of \`apply(print, Arguments(io, xs))\`.
> 
> \### \`splattable\`
> 
> \`splattable\` may be used to solve performance problem discussed in #29114:
> 
> \`\`\`julia
> splattable(x::Number) = (x,)
> splattable(x::StaticArray) = Tuple(x)
> \`\`\`
> 
> In case of \`Broadcasted\`, it can be used for calling \`instantiate\`:
> 
> \`\`\`julia
> splattable(x::Broadcasted) = instantiate(x)
> \`\`\`

Spoiler: @​JeffBezanson has a convincing argument against this direction so I guess it is not likely to happen.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 2, 2019, 4:15am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/17 "2019-12-02T04:15:15Z")

</div>

> [@tkf](#):
>
> Why not

Very cute approach. But it’s the same? I have to overwrite every function with `GhostFunction` also, what if `a` doesn’t appear in the first position?

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [December 2, 2019, 4:42am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/18 "2019-12-02T04:42:40Z")

</div>

If there are _n_ definitions like `fn(a::GhostArray, kwargs...) = fn(materialize(a), kwargs...)` for _m_ functions like `fn`, `GhostFunction`-based approach requires _O(n) + O(m)_ lines; you need _O(nm)_ lines in a naive approach.

> [@xiaodai](#):
>
> what if `a` doesn’t appear in the first position

You can use `map` over `args` and `materialize` it if the argument `isa GhostArray`. Julia is clever enough to compile it away.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 2, 2019, 6:51am UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/19 "2019-12-02T06:51:47Z")

</div>

> [@xiaodai](#):
>
> So I want for all `fn` , do the above. But there is no way to know all possible `fn` . And people can always define more `fn` s.

This question comes up very frequently. I think that the right solution to this problem is to define and document an _interface_ that a particular thing has to comply with, and keep it small.

If you just want to implement an [`<:AbstractArray`](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-array-1), I am not sure where those “hundreds ot methods” are coming from. Ideally, if they work for an `<: AbstractArray`, and you implement the API for the latter, they should work for your type, too.

---

<div class="post-metadata">

**Author:** ![longemen3000](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/longemen3000/32/7298_2.png) [@longemen3000](https://discourse.julialang.org/u/longemen3000)\
**Post date:** [December 2, 2019, 7:04pm UTC](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731/20 "2019-12-02T19:04:04Z")

</div>

One option is the use of a funxtion thst dispatches on that argument:

```julia
getvalue(a) =a
getvalue(a::GhostArray) = materialize(a) 

```

And if the defined methods are yours, then you just use `getvalue` everywhere

[Next page](https://discourse.julialang.org/t/trojan-horse-types-a-type-that-takes-over-the-function-calling-it/31731.md?page=2)