Dotdot: The double-broadcast operator enabling Float32..(a)

Internals & Design
1

Programming Julia for years, I have always missed a way to use double-broadcasting.
Ref is preventing a broadcast, but often one wants to also apply a broadcast operation to each element of an iterable. This can be done by encapsulating one of the two broadcasts in an (anonymous) function:

julia> a = [reshape(1:4,(2,2)) for _=1:3]; broadcast((x)->Float32.(x), a)
3-element Vector{Matrix{Float32}}:
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]

But this feels awkward and one often wishes to have access to a .. operator.
So I had a go at it:

function (..)(f, nargs...)
    mydot(nargs2...)=f.(nargs2...)
    mydot.(nargs...)
end

a = [reshape(1:4,(2,2)) for _=1:3]; Float32..(a)
3-element Vector{Matrix{Float32}}:
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]

I did not quite expected that this ends up to be that simple, but for some (great!) unknown reason the .. notation automatically worked in infix notation.

Can we not use this implementation as a general addition to the Julia Language? I am more than happy to write a PR (with help-file and tests), if agreed.
Of course there are many more applications than just this casting every element of arrays in a collection. For example adding a Tuple to each array of Tuples is another common use-case.

julia> a = [(1,2),(3,4),(5,6)]; ..(+,a,Ref((3,3)))
3-element Vector{Tuple{Int64, Int64}}:
 (4, 5)
 (6, 7)
 (8, 9)

Maybe there is even a way to enable the notation a ..+ Ref((3,3))?
What do you think?

See also this somewhat related thread:

[why doesn't `@.` always broadcast?]

2

there are a number of places where .. is already being used for some kind of a range (it’s the same precedence as :), like uniform distributions in MonteCarloMeasurements.

we already have a number of operators which can’t even use a single dot (:= $= ?: in isa : .. $ :: . ' ... -> ,) because it would be to easy to confuse it with something else, and ..$ notation would only make this problem worse. if anything, I would prefer to see the second dot added on top, like :$, but that’s already notation for creating a symbol.

often, I find that when I start wanting a second distribution, it’s really a better idea to just bring one layer out into an explicit loop which is more readable anyways:
[Float32.(A) for A=a] and
[A.+r for A=a, r=Ref((3,3))] or Ref(A .+ r for A=a, r=Ref((3,3)))

1 Like
3

You can use a macro @.., akin to @..

julia> macro (..)(args)
           args
       end
@.. (macro with 1 method)

julia> @.. 1+2
3
4

To add, someone is going to ask how to do triple broadcasting, and ... is taken. There’s nothing particular about double broadcasting to stop there. Julia does have Base.BroadcastFunction(op) to represent a broadcasted operator. We can just nest those, though I’d rather not write that repeatedly.

julia> fiter(f, n) = reduce(∘, Iterators.repeated(f, n))
fiter (generic function with 1 method)

julia> bcn(op, n) = fiter(Base.BroadcastFunction, n)(op)
bcn (generic function with 1 method)

julia> bcn(Float32, 2)
Base.Broadcast.BroadcastFunction(Base.Broadcast.BroadcastFunction(Float32))

julia> a = [reshape(1:4,(2,2)) for _=1:3]; bcn(Float32, 2)(a)
3-element Vector{Matrix{Float32}}:
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]

Repeated function composition is not type-stable because the type depends on the runtime number of calls. The straightforward Val tweak doesn’t work, the type inference basically stops after the first composition. If that could be made type-stable, that’d be a nice package or addition to Julia.

A macro could also just paste the code in because they do take Int literals. It looks real odd though, not nearly as neat as overhauling the parser for even more dots:

julia> macro ..(n::Int, op)
         ex = :(Base.BroadcastFunction($op))
         for _ in 2:n
           ex = :(Base.BroadcastFunction($ex))
         end
         esc(ex)
       end
@.. (macro with 1 method)

julia> @..(2, Float32)
Base.Broadcast.BroadcastFunction(Base.Broadcast.BroadcastFunction(Float32))

julia> a = [reshape(1:4,(2,2)) for _=1:3]; @..(2, Float32)(a)
3-element Vector{Matrix{Float32}}:
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
1 Like
5

I can see that point. But practically speaking, I have “needed” the double broadcast many times, but I never needed the triple broadcast as far as I remember.

6

not sure I get it. The macro seems to do nothing. Or am I missing a point here?

7

I guess this is real problem, if there are potential name clashes. But even a normal name such as dotdot would be helpful to provide this functionality.

8

If you just need double, an alias for Base.BroadcastFunction would help:

julia> const bcf = Base.BroadcastFunction
Base.Broadcast.BroadcastFunction

julia> a = [reshape(1:4,(2,2)) for _=1:3]; bcf(Float32).(a)
3-element Vector{Matrix{Float32}}:
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]

That alias has a risk of clashing with another bcf the same way .. clashes, though I think the risk is lower in practice. The parser would have to be overhauled for an infix syntax either way, which would be too far for just double broadcasting.

9

technically you can also do this with the postfix unary operator:

julia> var"'²"(a) = Base.BroadcastFunction(Base.BroadcastFunction(a))
'² (generic function with 1 method)

julia> Float32'²(a)
3-element Vector{Matrix{Float32}}:
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]
 [1.0 3.0; 2.0 4.0]

but the point I was making was that if you start using a double-broadcast all over the place, you’ll pretty quickly start feeling the need for a third and so on, and there’s a surprising number of edge cases that pop up where behavior is slightly different than you would expect it to be. (speaking as the person who started that thread you linked there)

I had something similar to

var"'²"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 2))(a)
var"'³"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 3))(a)
var"'⁴"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 4))(a)
var"'⁵"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 5))(a)
var"'⁶"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 6))(a)
var"'⁷"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 7))(a)
var"'⁸"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 8))(a)
var"'⁹"(a) = reduce(∘, Iterators.repeated(Base.BroadcastFunction, 9))(a)

in my startup file at one point, and while it was fun for a bit, it got confusing really fast
(actually I think I had it recursively defined and was doing various other dumb things as well, maybe these 'ⁿ functions would end up being nicer).

1 Like
10

Yeah, sorry, that’s a no-op there. My point is just that this is available syntax for you to define to do whatever you wanted :slight_smile: