# Package for lazy hcat/vcat of a large number of vectors

**URL:** <https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590>\
**Category:** General Usage\
**Tags:** question\
**Created:** [August 31, 2022, 10:50am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590 "2022-08-31T10:50:57Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 31, 2022, 10:50am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/1 "2022-08-31T10:50:57Z")

</div>

I am looking for the lazy equivalent of

```julia
v = [rand(500) for _ in 1:5000];
mapreduce(permutedims, vcat, v)

```

where both dimensions may be large (up 10⁵). This is trivial to code up but I want to avoid duplication.

Is there a package which has such functionality?

I thought of LazyArrays.jl but it only has vararg syntax, cf

> <https://github.com/JuliaArrays/LazyArrays.jl/issues/97>
>
> The current way of constructing \`Vcat\` objects, by splatting/slurping, causes st…ack overflow when concatenating a large number of arrays since special methods of these functions are compiled for the particular number of input arguments.
> 
> My usecase is LazyWAVFiles.jl which treats a folder of WAV files as one large array, such a folder may contain quite a lot of files...

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [August 31, 2022, 10:56am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/2 "2022-08-31T10:56:12Z")

</div>

This one from RecursiveArrayTools usually works quite well:

[https://recursivearraytools.sciml.ai/stable/array\_types/#RecursiveArrayTools.VectorOfArray](https://recursivearraytools.sciml.ai/stable/array_types/#RecursiveArrayTools.VectorOfArray)

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 31, 2022, 10:57am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/3 "2022-08-31T10:57:19Z")

</div>

Thanks! After I pressed submit, I also realized that

```julia
using JuliennedArrays
Align(v, False(), True())

```

works too.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 31, 2022, 11:04am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/4 "2022-08-31T11:04:36Z")

</div>

SplitApplyCombine.jl has functions specifically for this, both lazy and eager.

```julia
julia> using SplitApplyCombine

# eager - combinedims
julia> combinedims(v)
500×5000 Matrix{Float64}:
...

julia> combinedims(v, 1)
5000×500 Matrix{Float64}:
...

# lazy - just change to combinedimsview

```

The inverse operation is there as well - `splitdims`.

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [August 31, 2022, 11:18am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/5 "2022-08-31T11:18:46Z")

</div>

You could also use TensorCast, which has the most intuitive syntax:

```julia
using TensorCast
@cast m[i,j] := v[i][j]  

```

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 31, 2022, 1:58pm UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/6 "2022-08-31T13:58:43Z")

</div>

This lead me to `LazyArrays.stack`, which I ended up using.

Thanks for all the great replies.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 31, 2022, 2:01pm UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/7 "2022-08-31T14:01:44Z")

</div>

A related question: what if the elements are matrices, as in

```julia
v = [rand(500, 50) for _ in 1:5];
reduce(vcat, v) # need lazy version

```

---

<div class="post-metadata">

**Author:** ![rafael.guerra](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rafael.guerra/32/216610_2.png) [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)\
**Post date:** [August 31, 2022, 2:22pm UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/8 "2022-08-31T14:22:21Z")

</div>

To combine two indices into one, you can use TensorCast’s operator ⊗:

```julia
@cast m[j⊗i, k] := v[i][j,k] (i in 1:5) 

```

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [August 31, 2022, 4:59pm UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/9 "2022-08-31T16:59:34Z")

</div>

There is also `SentinelArrays.ChainedVector` type specifically for vectors. It uses a vector internally, so won’t have the same StackOverflow problem.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [September 1, 2022, 10:11am UTC](https://discourse.julialang.org/t/package-for-lazy-hcat-vcat-of-a-large-number-of-vectors/86590/10 "2022-09-01T10:11:13Z")

</div>

> [@Tamas\_Papp](#):
>
> what if the elements are matrices

Exactly the same solution with `combinedimsview` (: