Destructuring and broadcast

General Usage
1

If I want to broadcast a function that returns a tuple, is there a simple way to “broadcast” the left hand side destructuring?

a = 1:10
b = 11:20

transform(a, b) = a+b, a*b, a-b # MWE, the real one is more compicated

## want something like (syntax of course does not work)
@. x, y, z = transform(a, b)

## of course I can do
w = transform.(a,b)
x = first.(w)
y = (v->v[2]).(w)
z = last.(w)
6 Likes
2

I don’t know about simple, but I had the same problem and came up with this oneliner for creating NTuples:

x,y,z = collect(zip(transform.(a,b)...))

If you absolutely need Vectors then add another collect:

x,y,z = collect.(collect(zip(transform.(a,b)...)))

But I’m still a newbie, so I’m sure there’s a better way. I’m curious to see what the pros come up with.

3

Hi,

I have a similar issue: I don’t know how to obtain a tuple of arrays instead of an array of tuples.

The answer by @NickNack works for simple cases, but loses track of resulting shape when the broadcast results in more than one dimension.

# compute on a 2d grid
x1 = reshape(1:5,  (5,1)) 
x2 = reshape(6:11, (1,6))
typeof(collect(zip(transform.(x1,x2)...))) # Array{NTuple{30,Int64},1}

This result is different from the solution by @Tamas_Papp, which correctly preserves the shape of the resulting arrays.

Coming from python/numpy, I am used to this situation resulting in tuple of arrays instead of array of tuples. Anybody knows a way to obtain similar behavior in julia?

The equivalent numpy code:

import numpy as np

def transform(a, b):
    return a+b, a*b, a-b

x1 = np.arange(1, 6).reshape((5,1))
x2 = np.arange(6, 12).reshape((1,6))
u, v, w = transform(x1, x2)

Potentially related:
https://github.com/JuliaLang/julia/issues/13942

None of the proposed solutions seem to work here, upon a very superficial quick try.

4

Relevant issue: https://github.com/JuliaLang/julia/issues/22129

5

In v0.6, I used StructsOfArrays to do broadcasting with tuple output:

using StructsOfArrays

T = Float64
TV = Vector{Float64}

a = zeros(T, 10); b = copy(a)
s = StructOfArrays{NTuple{2,T}, 1, Tuple{TV, TV}}((a, b))

f(a, b) = (a + b, a * b)

s .= f.(1:10, 1:10)

A macro for the above should be simple enough. Figuring out T and TV and handling the case where a and b are not already allocated are the main challenges I can see.

2 Likes
6

Why not use dot-operations inside a function? As far as I can see, it works as well for scalars.

a_ = collect(a)
b_ = collect(b)
transform(a, b) = a .+ b, a .* b, a .- b

x, y, z = transform(a_, b_)

Also, if there are only +, - and * operations, they probably can be redefined as dot-wise for properly-sized AbstractArrays ?

7

Assume that the transformation is a black box, eg because it involves a costly, large intermediate result that you don’t want to allocate. Eg

function transform(a, b)
    m = g(a, b)
    make_x(m), make_y(m)
end

where g is costly and m is large.

8

It seems that four years after I am finding the exact same issue. Is there a better or preferred solution now?

9

LazyRow + StructArray would be my go to solution now.

https://github.com/JuliaArrays/StructArrays.jl

1 Like
10

Thanks, @Tamas_Papp I have no previous experience with StructArrays. I expected something like this to work:

julia> using StructArrays

julia> a = 1:10
1:10

julia> b = 11:20
11:20

julia> transform(a, b) = a+b, a*b, a-b
transform (generic function with 1 method)

x, y, z = StructArray(transform.(a, b))
10-element StructArray(::Vector{Int64}, ::Vector{Int64}, ::Vector{Int64}) with eltype Tuple{Int64, Int64, Int64}:
 (12, 11, -10)
 (14, 24, -10)
 (16, 39, -10)
 (18, 56, -10)
 (20, 75, -10)
 (22, 96, -10)
 (24, 119, -10)
 (26, 144, -10)
 (28, 171, -10)
 (30, 200, -10)

julia> x    # Get the first row here instead of first column (which I want)
(12, 11, -10)

The only option I can find now is:

julia> w = StructArray(transform.(a, b))
10-element StructArray(::Vector{Int64}, ::Vector{Int64}, ::Vector{Int64}) with eltype Tuple{Int64, Int64, Int64}:
 (12, 11, -10)
 (14, 24, -10)
 (16, 39, -10)
 (18, 56, -10)
 (20, 75, -10)
 (22, 96, -10)
 (24, 119, -10)
 (26, 144, -10)
 (28, 171, -10)
 (30, 200, -10)

julia> x = getproperty(w, 1)
10-element Vector{Int64}:
 12
 14
 16
 18
 20
 22
 24
 26
 28
 30

Which is not so different from this approach without StructArrays:

julia> w = transform.(a, b)
10-element Vector{Tuple{Int64, Int64, Int64}}:
 (12, 11, -10)
 (14, 24, -10)
 (16, 39, -10)
 (18, 56, -10)
 (20, 75, -10)
 (22, 96, -10)
 (24, 119, -10)
 (26, 144, -10)
 (28, 171, -10)
 (30, 200, -10)

julia> x = getindex.(w, 1)
10-element Vector{Int64}:
 12
 14
 16
 18
 20
 22
 24
 26
 28
 30

Am I missing something? What I am looking for is a “direct way” to unpack the columns of a broadcasted function. Something like @. x, y, z = transform(a, b) which you mentioned at the beginning of the this post.

11

You can use components():

using StructArrays
a, b = 1:10, 11:20
transform(a, b) = a+b, a*b, a-b
w = StructArray(transform.(a, b))
x, y, z = StructArrays.components(w)
2 Likes
12

I asked a related question at the Tables.jl repo. It looks like the compiler should be smart enough to make this work even if you have an intermediate collection of tuples. Does Tables have the infrastructure for a performant map that returns vectors of tuples? · Issue #237 · JuliaData/Tables.jl · GitHub

1 Like
13

As a tiny improvement, one can also do

w = StructArray(Iterators.map(transform, a, b))

so that the intermediate vector of structs is not created at all, but one only allocates the columns that will be used later. I think a similar trick can be done with broadcast, it should probably be something like

w = StructArray(Broadcast.Broadcasted(transform, (a, b)))

This could maybe even justify defining a small helper function in StructArrays, like

unzip(iter) = components(StructArray(iter))
unzipwith(f, args...) = unzip(Iterators.map(f, args...)) # is this the correct name?

though it should probably not be exported, because the name is very generic.

5 Likes