# Generic and flexible array-of-struct to struct-of-array conversions

**URL:** <https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420>\
**Category:** General Usage\
**Tags:** data\_structures, arrays\
**Created:** [December 21, 2021, 7:28am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420 "2021-12-21T07:28:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 21, 2021, 7:28am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/1 "2021-12-21T07:28:03Z")

</div>

I have a problem that comes up fairly often that I do not really have a satisfying solution to at the moment.  
Say I have a container type like the one below, representing many “versions” of a real number

```julia
struct Baz{T} <: Real
    a::Vector{T}
end

```

and that I can have arrays of such containers, for instance the following complex version

```julia
R = [Complex(Baz(randn(3)), Baz(randn(3))) for i in 1:2, j in 1:2]

julia> typeof(R)
Matrix{Complex{Baz{Float64}}}

```

When performing some operations on `R`, I would essentially like to permute the memory layout of `R` by unpacking the elements of `Baz.a` one by one so that I get `r` illustrated by the pseudo code below, perform some operation on `r` and then repackage the result back into the structure `Baz` again.

```julia
function eigvals(R::Matrix{Complex{Baz{T}}}) where T
    # Allocate E
    for all k
        r[i,j] = Complex(R[i,j].re.a[k], R[i,j].im.a[k]) # "unpack R into r"
        e = eigvals(r) # operate on r
        E[i].re.a[k] = e[i].re # repackage result into Baz
        E[i].im.a[k] = e[i].im
    end
    E
end

```

So far, this can be done manually without too much of a problem, but there might be variations of the problem, for instance, if instead of `eigvals` I call `svdvals`, the result is a vector of real values rather than complex, and I would like to do something like this instead:

```julia
function svdvals(R::Matrix{Complex{Baz{T}}}) where T
    for all k
        r[i,j] = Complex(R[i,j].re.a[k], R[i,j].im.a[k])
        s = svdvals(r)
        S[i].a[k] = e[i]
    end
    S
end

```

The data layout of `Baz` is chosen to fit well for scalar operations, but when `Baz` appears in more complicated contexts, I lack a general solution that doesn’t require customizing the implementation for each function I want to handle, i.e., `eigvals, svdvals` above. I’ve been looking towards [Accessors.jl](https://juliaobjects.github.io/Accessors.jl/dev/) which could potentially serve as a piece of the puzzle, but I can’t really find something that does exactly what I need. For completeness, below is a manual implementation of eigvals that does what I want

```julia
using LinearAlgebra
struct Baz{T} <: Real
    a::Vector{T}
end

Base.show(io::IO, baz::Complex{Baz{T}}) where T = print(io, "Complex{Baz{$T}[($(baz.re.a), $(baz.im.a))")

function LinearAlgebra.eigvals(R::Matrix{Complex{Baz{T}}}) where T
    K = length(R[1].re.a)
    # Allocate E
    E = similar(R, size(R,1))
    for i in eachindex(E)
        E[i] = Complex(Baz(zeros(K)), Baz(zeros(K)))
    end
    r = zeros(Complex{T}, size(R)...)
    for k in 1:K
        for j in eachindex(R)
            r[j] = Complex(R[j].re.a[k], R[j].im.a[k]) # "unpack R into r"
        end
        e = eigvals(r) # operate on r
        for i in eachindex(e)
            E[i].re.a[k] = e[i].re # repackage result into Baz
            E[i].im.a[k] = e[i].im
        end
    end
    E
end

R = [Complex(Baz(randn(3)), Baz(randn(3))) for i in 1:2, j in 1:2]
E = eigvals(R)

```

Generally, the array `R` may have any number of dimensions and produce a result of any number of dimensions, and both `R` and `E` may be arrays of either complex or real versions of `Baz`.  
Any idea that would help in implementing functions like the above would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![FMeirinhos](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fmeirinhos/32/19527_2.png) [@FMeirinhos](https://discourse.julialang.org/u/FMeirinhos)\
**Post date:** [December 21, 2021, 10:04am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/2 "2021-12-21T10:04:09Z")

</div>

Have you looked into [StructArrays.jl](https://github.com/JuliaArrays/StructArrays.jl)? Even though the package has always mystified me, I believe that some accessor magic can be worked out.

Consider your matrix `R` in the following form

```julia
R = let
    R = [Complex(Baz(randn(3)), Baz(randn(3))) for i in 1:2, j in 1:2]
    StructArray(R; unwrap = t -> t <: Baz)
end;

```

Then you can easily access the imaginary / real part like so `S.re.a` or `S.im.b`. I understand this is still far from solving your problem but may shine some light onto the problem.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 21, 2021, 10:24am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/3 "2021-12-21T10:24:24Z")

</div>

Thank you @FMeirinhos ! I have looked at the docs of StructArrays but got a bit confused by all the options 🙂 Thank you for showing me this trick, it can certainly simplify writing these custom functions.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 21, 2021, 10:28am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/4 "2021-12-21T10:28:14Z")

</div>

It occured to me that if `isbitstype(Baz)` I could use reinterpret to get an array which I could easily do permutedims on and then operate using `mapslices` etc., but due to the vector in `Baz`, thsi is not possible. Perhaps there’s a tool, maybe StructArrays is this tool, that could provide a `flatten/reinterpret` that is flexible enough to handle `R` in the example?

---

<div class="post-metadata">

**Author:** ![FMeirinhos](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fmeirinhos/32/19527_2.png) [@FMeirinhos](https://discourse.julialang.org/u/FMeirinhos)\
**Post date:** [December 21, 2021, 10:35am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/5 "2021-12-21T10:35:27Z")

</div>

Undoubtedly the problem is screaming _lenses_ but I am not sure in practical terms (i.e., finite amount of developing time) you can use them… Hopefully I am proven wrong!

I would say that the absolute easiest thing you can do is to have the complex type at the lowest level `Baz{Complex{T}}` instead of `Complex{Baz{T}}`, since that way you could leverage the algebraic rules / functions of a complex `T` instead of complex `Baz{T}`. But I guess that’s not feasible for your use cases. But it would make all of these custom multiple dispatches essentially trivial.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [December 21, 2021, 10:45am UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/6 "2021-12-21T10:45:56Z")

</div>

The problem of choosing `T{S}` vs. `S{T]` where both `T` and `S` are `<: Number` is something I encounter quite often, and I’ve come to the conclusion that you want both representation in different contexts. This arises when, for instance, you combine any of the following, `Unitful.Quantity, Complex, ForwardDiff.Dual, MonteCarloMeasurements.Particles` and the list goes on…  
Implementing functionality with method overloading only goes so far when it comes to combining special number types like this. It’s primarily the outermost type that determines dispatch, so, to make a number behave like `Complex`, `Complex` has to be the outermost type.

If I were to have `Baz{Complex{T}}`, it would not make sense to define `Baz <: Real`, and I’m not allowed to define `Baz{T} <: T`, so then I’m left with either defining `Baz <: Number` or having a separate `ComplexBaz`. There are many hacky workarounds to this problem, but they all fell like just that, hacky workarounds 😕

---

<div class="post-metadata">

**Author:** ![FMeirinhos](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fmeirinhos/32/19527_2.png) [@FMeirinhos](https://discourse.julialang.org/u/FMeirinhos)\
**Post date:** [December 21, 2021, 2:39pm UTC](https://discourse.julialang.org/t/generic-and-flexible-array-of-struct-to-struct-of-array-conversions/73420/7 "2021-12-21T14:39:09Z")

</div>

Ohhh, I do feel you. I think I have lost the war against `Dual` `Complex`s a long time ago…

Related with your original problem, possibly [`LazyStack.jl`](https://github.com/mcabbott/LazyStack.jl) could be used in tandem

```julia
using StructArrays
using LazyStack

C = let
    C = [Complex(Baz(randn(3)), Baz(randn(3))) for i in 1:2, j in 1:2]
    C = StructArray(C; unwrap = t -> t <: Baz)
end;

C_re = stack(C.re.a)
C_im = stack(C.im.a)

R = let
    R = [Baz(randn(3)) for i in 1:2, j in 1:2]
    StructArray(R; unwrap = t -> t <: Baz)
end;

R_ = stack(R.a)

```