Flatten to a vector

Sorry @Raf, I can’t seem to find the documentation for this simple question:

Is there a way for Flatten.flatten to return a Vector instead of a Tuple of the parameters?

Right now I simply do:

[flatten(mystruct)...]

Thanks!

2 Likes

I usually use reduce(vcat,A) but never timed it. Let’s do it!

julia> using Base.Iterators

julia> A = map(i->rand(10),1:1000);

julia> @btime reduce(vcat,A);
  10.179 μs (2 allocations: 78.20 KiB)

julia> @btime [flatten(A)...];
  639.579 μs (30014 allocations: 1019.94 KiB)
3 Likes

That’s cool, but I’m trying to flatten a struct, not a vector of vectors. Actually, a vector of structs…

Flatten isn’t for vectors - its all compile time and we don’t know vector length at compile time.

It flattens arbitrary nested structs and tuples where the fields are known at compile time.

Edit: you are actually timing Base.Iterators.flatten, not Flatten.jl. You would usually find that flattening structs at runtime is best measured in ns. It might even have a similar runtime fro a problem as large as the one you time above if it was using tuples, but the compile time would be ridiculous.

There used to be an argument or flattening to Vector but as I added all the arbitrary type flattening it got confusing having so many types in the method call.

So now you just manually do [flatten(x, Real)...]. That’s all the internal method did anyway and its actually less characters than it was before :slight_smile:

I should probably update the docs a little.

Edit: you might also want to convert the tuple to SVector where performance matters, as it wont allocate.

2 Likes

Thanks!

In my application I want to flatten a known number of structs to an iterable container. Each struct has 4 fields (UInt8), and there are 4 structs, which would result in a tuple with 16 (UInt8) elements. So an SVector would make sense. So something like this (on my phone now, so will check my self later)?

x = [MyStruct() for _ in 1:4]
flatten(x..., SVector{UInt8, 16})

(back on computer) Apparently not… I’m all of a sudden unsure on how to do this with Flatten… I could forgo my custom type and just use a static vector:

using StaticArrays
A = SVector{4, UInt8}
x = (A(rand(UInt8, 4)) for _ in 1:4)
reduce(vcat, x)

How exactly, and would it be any better, do I flatten x with Flatten:

using Flatten
struct B
    x1::UInt8
    x2::UInt8
    x3::UInt8
    x4::UInt8
end
x = (B(rand(UInt8, 4)...) for _ in 1:4)

You could look at https://github.com/pao/StrPack.jl. Not sure how well it works since it seems to not have been extensively maintained.

Cant you just do SVector(flatten(x, Real)) ? (or whatever type you want besides Real)

The type in the flatten signature is the types that will be extracted from the struct! So flatten(x, Int) will get all the Int and SVector(flatten(x, Int)) will put all the Int into an SVector.

Flatten args are like: flatten(obj, extracttype::Type, ignoretype::Type), returning a tuple where extract is the type/union to take and ignore is the type/union to ignore. It’s totally arbitrary what actually gets flattened now, but it always produces a tuple. Then you can do whatever you want with the tuple.

Flatten.jl does a lot more than StrPack and is maintained. It’s just very abstract and kinda weird.

1 Like

Sorry, it was unclear to me that Flatten here referred to a package.

No, because x in this example is a vector of structs. But I might be missing something. In this MWE, that won’t work:

using Flatten, StaticArrays
struct B
    x1::UInt8
    x2::UInt8
    x3::UInt8
    x4::UInt8
end
x = [B(rand(UInt8, 4)...) for _ in 1:4]

You can’t flatten vectors as the length is unknown to the compiler. Try with a tuple or SVector:

using Flatten
struct B
    x1::UInt8
    x2::UInt8
    x3::UInt8
    x4::UInt8
end

julia>  x = Tuple(B(rand(UInt8, 4)...) for _ in 1:4)

julia> using StaticArrays

julia> SVector(flatten(x))
16-element SArray{Tuple{16},UInt8,1,16} with indices SOneTo(16):
 0x81
 0xa4
 0x41
 0xef
 0xf6
 0xca
 0x9b
 0x7d
 0xb6
 0xe7
 0xb4
 0xd7
 0x69
 0x14
 0xf7
 0x97
1 Like

Also of interest:

julia> @btime SVector(flatten($x))
  0.020 ns (0 allocations: 0 bytes)
16-element SArray{Tuple{16},UInt8,1,16} with indices SOneTo(16):
 0x81
 0xa4
 0x41
 0xef
 0xf6
 0xca
 0x9b
 0x7d
 0xb6
 0xe7
 0xb4
 0xd7
 0x69
 0x14
 0xf7
 0x97

Thank you so much @Raf, this is great. I was trying to (B(...) for _ in 1:4) before instead of the explicit Tuple(...) version and got stuck on that.

1 Like

It good to know what is confusing about the package, I’ll make it clearer in the docs that length of objects has to be known at compile time, and we are specifically flattening fields, which Array doesn’t have.

2 Likes

I have been using vcat(a, b..., c, d) etc for this (in the vector case). Would have liked to see a proper flatten though, as well as flat_map.

I think it could be done for a struct by using the dynamic lookup functions.

flatten_struct = x -> [getfield(x, n) for n = fieldnames(typeof(x))]
julia> using FlexiMaps

julia> flatten([1:2, 5:10, 1:0])
8-element Vector{Int64}:
<...>

julia> flatmap(i -> 1:i, [1, 3, 2])
6-element Vector{Int64}:
 1
 1
 2
 3
 1
 2

This does a single level of struct flattening. It can be what one needs, but Flatten.jl main usecase is recursive flattening AFAIK.

in this example.
How does flatten() stop unnesting at the rational number level and not unpack the numerators and denominators as well?
I’d be interested in understanding it in principle as well, without the implementation details that I probably wouldn’t be able to understand.

using Flatten

struct NR
    q1::Rational{Int64}
    q2::Rational{Int64}
end

NSR = Tuple(NR(nn...) for _ in 1:4)

nn=Tuple(NR((Rational(rand(1:10,2)...) for _ in 1:2)...) for _ in 1:4)
julia>         flatten(nn)
(5//2, 2//1, 2//1, 7//9, 2//3, 3//5, 9//5, 3//5)

This is just because you are not passing any arguments to flatten, and the default behavior is to flatten Real. Maybe that’s not well enough documented.

If you want the inner values, use flatten(nn, Int64) and it will look inside the Rational numbers for Int64.

1 Like

Okay. Thank you.
I don’t need to unpack the rationali.
I was wondering how the function could be so “smart” that it stops at the right point.
So, conceptually, the recursive function (with default parameters) unpacks everything that isn’t Real?