Map a vector to multiple vectors

General Usage
1

Is there a simple way to map a vector to multiple vectors?

vin = [1,2,3,4]
v1, v2, v3 = map(v) do x
  return x, 2x, 3x
end

The above doesn’t work, but that shows what I mean. Usually, you’d get a vector of tuples from the above, but is there an idiomatic way to get arrays directly instead?

I know that I could do something like:

vin = [1,2,3,4]
v1 = Vector{Int}(undef, length(vin))
v2 = Vector{Int}(undef, length(vin))
v3 = Vector{Int}(undef, length(vin))
for i in eachindex(vin)
  v1[i] = x
  v2[i] = 2x
  v3[i] = 3x
end

so I’m asking if there is an existing more concise way.

Note: I would prefer a solution that is at least as performant as that. For example, doing 3 separate map/broadcasts which seems to be about 25% slower for large arrays (which is what I’m working with).

Performance test for 3 approaches 
div2(x) = x / 2
times2(x) = 2*x
times3(x) = 3*x

function test1(v)
    return map(div2, v), map(times2, v), map(times3, v)
end

function test2(v)
    return div2.(v), times2.(v), times3.(v)
end

function test3(v)
    len = length(v)
    v1 = Vector{Float64}(undef, len)
    v2 = Vector{Int}(undef, len)
    v3 = Vector{Int}(undef, len)
    @inbounds for i in eachindex(v)
        x = v[i]
        v1[i] = div2(x)
        v2[i] = times2(x)
        v3[i] = times3(x)
    end
    return v1, v2, v3
end

using BenchmarkTools
function time3(n=100000)
    v = fill(17, n)
    @btime test1($v)
    @btime test2($v)
    @btime test3($v)
    return
end
2

I haven’t used it, but Unzip.jl might be of use to you!

1 Like
3

Here’s an old snippet I’ve used before for repacking an array of tuples into a tuple of arrays

function repack(x::AbstractArray{T}) where T<:Tuple
	# turn Array{Tuple{T1,T2...}} to Tuple{Array{T1},Array{T2},...}
	fT = ntuple(i->fieldtype(T,i),fieldcount(T)) # fieldtypes(T) would be the obvious choice but is type-unstable
	arrs = similar.(Ref(x),fT)
	for i in eachindex(x)
		@inbounds setindex!.(arrs,x[i],Ref(i))
	end
	return arrs
end
function repack(x::AbstractArray{NamedTuple{N,T}}) where {N,T<:Tuple}
	# turn Array{NamedTuple{N,Tuple{T1,T2...}}} to NamedTuple{N,Tuple{Array{T1},Array{T2},...}}
	fT = ntuple(i->fieldtype(T,i),fieldcount(T)) # fieldtypes(T) would be the obvious choice but is type-unstable
	arrs = similar.(Ref(x),fT)
	for i in eachindex(x)
		@inbounds setindex!.(arrs,Tuple(x[i]),Ref(i))
	end
	return NamedTuple{N}(arrs)
end

Unfortunately, it’s only doing the processing after the map is complete so does impose a little extra overhead, which may be relevant when the map is quick and performance is important. Perhaps there is a cleaner solution using StructArrays.jl?

You could also look into LazyArrays.jl or use something like

v1 = Broadcast.Broadcasted(tup -> tup[1], (array_of_tuples,))

to make lazy array-like wrappers over the single output of map, depending on your precise use case.

4
  • If you already did
mv = map(v) do x
  return x, 2x, 3x
end

and want separate (abstract)vectors:

using FlexiMaps
v1 = mapview(1, mv)
v2 = mapview(2, mv)
...

This doesn’t copy or allocate anything new.

  • If you just have v and want these three vectors, a more direct way is:
using StructArrays

mv = map(StructArray(_=vin)) do x
    x._, 2*x._, 3*x._
end

# this is free - doesn't allocate anything:
v1 = mv.:1

or simply use mv.:1 in place of v1. Unlike the first scenario with mapview, these are actual Vectors not just AbstractVectors,

If you seek efficiency and your original vector contains some structs, not just Ints, then try making it a StructArray from the beginning.

1 Like
5

Sorry for the too basic question, but I do not understand why not just:

v1, v2, v3 = vin, 2vin, 3vin
6

That was just a trivial example to illustrate. My real use case is more complex.

7

Thank you for the suggestions. I put the note in about performance because I didn’t want to do post repacking. I expect doing the map/broadcast separately would be faster than repacking anyway. And for the view suggestions, that would work for some cases, but not others. For example, putting the data into another datastructure where it would reuse the memory if it was a vector, but it will allocate otherwise, in which case the view doesn’t help.

So, I’m gathering that the answer to my question is no, it doesn’t exist. Which is fine, I just wanted to know. I realized there is a challenge regarding type stability because the vector eltype isn’t known until you run the function at least once. But I think the implementation of map has a way to solve that, so one would just have to do it the same way.

8

Could you map the transformations instead:

F = [x->x, x->2x, x->3x]
vin = [1,2,3,4]
v1, v2, v3 = map(F) do f
  f(vin)
end
2 Likes
9

I’m pretty sure your concerns about performance should be solved by the StructArrays approach.

10

Hard to know if that applies to your problem, but this is quite concise:

julia> map.((div2,times2,times3), Ref(vin))
([0.5, 1.0, 1.5, 2.0], [2, 4, 6, 8], [3, 6, 9, 12])

It doesn’t seem to perform worse than the other alternatives.

1 Like
11

I assume that (in the actual use case) the multiple outputs of this map share calculations, which is why they’re being computed together rather than with multiple separate map calls. Although if they’re complicated enough that this matters, the repacking (or lazy views) afterward are probably of minimal cost. That’s been my experience when encountering this problem in the wild.

Note that the above suggestion using Broadcast.Broadcasted created a lazy wrapper over the array-of-tuples, rather than materializing a result. It’s basically free. The LazyArrays package is a way to do this that doesn’t rely on semi-internal functionality, if you want a more robust solution. The only reason these wouldn’t work is if you need a certain memory layout to pass these to some external library.

But I think StructArrays addresses your case directly.

1 Like
12

Yes, these two points are important, and I should have made that more clear in my original post. That said, the replies have been very educational, and I appreciate them. Especially because, StructArrays does exactly what I’m looking for. The StructArray example in a previous post didn’t compile for me because map’ing a StructArray returned a Vector and not a StructArray. However, there is collect_structarray. Although I don’t know if there is a single method, it’s trivial to create one, so here it is:

function maparray(f, v)
    sa = StructArrays.collect_structarray(f(x) for x in v)
    return StructArrays.components(sa)
end

function test6(v)
    return maparray(v) do x
        (div2(x), times2(x), times3(x))
    end
end

It is a little slower than the “ideal” one (explicitly allocate and loop), but as the length of the vector gets larger, it approaches similar performance. The allocations and return types are the same.

Thank you all!

13

That’s strange… I just ran the code

and it did return a StructArray as expected.

1 Like
14

and it did return a StructArray as expected.

Sorry, it seems I have StructArray-0.5.1 because some other package is blocking it from upgrading. I ran it in a separate env with version 0.6.15 and it runs successfully. Thanks again!