Flatten to a vector

General Usage
, ,
21

Yeah, by default it stops at any Real and recurses over everything else.

The idea is you customise the use and ignore arguments to match your objects. These can be Union types if you need something complicated. Anything in use is taken as-is, anything in ignore is not recursed at all. Everything else is flattened until a use or ignore type is hit or there are no more child objects.

See: Flatten.jl/Flatten.jl at master · rafaqz/Flatten.jl · GitHub

2 Likes
23

Tanks!
I tried following the code in this simple case,

using Flatten
struct Foo{A,B,C}
           a::A
           b::B
           c::C
       end

nested = Foo(Foo(1, 2, 3), 4.0, 5.0)
#Foo{Foo{Int64,Int64,Int64},Float64,Float64}(Foo{Int64,Int64,Int64}(1, 2, 3), 4.0, 5.0)
flatten(nested)
#(1, 2, 3, 4.0, 5.0)

using the debugger in vscode, but when executing the function _flatten(x, ft, use, ignore) #line 132 everything crashes with the following error

┌ Error: Some Julia code in the VS Code extension crashed
└ @ VSCodeDebugger c:\Users\sprmn\.vscode\extensions\julialang.language-julia-1.38.2\scripts\error_handler.jl:15
ERROR: Method is @generated; try `code_lowered` instead.
Stacktrace:
 [1] error(s::String)
...

cursor stops at line 110 flatten_builder(T, fname) = quote
I assume the sequence is:

_flatten(x, ft, use, ignore)

#then
@generated _flatten(obj, flattentrait, use, ignore) = flatten_inner(obj)

#then
flatten_inner(T) = nested(T, flatten_builder, flatten_combiner)

#then
flatten_builder(T, fname) = quote
    if flattentrait($T, Val{$(QuoteNode(fname))})
        flatten(getfield(obj, $(QuoteNode(fname))), flattentrait, use, ignore)
    else
        ()
    end
end

but i’m not sure i understand, because the code is too complex for me, especially for the parts where metaprogramming is used.

Would it help me to understand something more if you could explain in a simple way, referring to the tested example, what function they have and what values they assume, the flattentrait, flatten_builder and flatten_combiner variables?

If it was this expression that causes problems for the debugger, how can I replace it with another equivalent but digestible by the debugger?

@generated _flatten(obj, flattentrait, use, ignore) = flatten_inner(obj)
24

but i’m not sure i understand, because the code is too complex for me, especially for the parts where metaprogramming is used.

You may be in for a rough time

Recursive functions are hard to understand, @generated functions are hard to understand and Flatten.jl is built on recursive generated functions.

There are also some complications because traits can be used to drop out fields with e.g FieldMetadata.jl. Wich is all macros that generate macros.

I would not write something like this any more. We had a cleaner version in Accessors.jl but can’t get it type stable.

Probably the debugger cant get through tany of this, you will just have to read it and puzzle.

Try to ignore reconstruct thats even worse.

25

But to answer your question a little the flatten_builder is building a quoted expression - its a functions that makes code. Here it needs the fiedname in a Val wrapper for the flattenable field trait. So it can all compile away.

nested is the generic recursive generated function walker all the other functions use.

flattentrait is any function like a FieldMetadata.jl trait that takes an object and fieldname, and returns a Bool.

The names are terrible, I apologise. It was my first year writing julia.

2 Likes
26

but does flatten only work for nested structures of the same type?

I tried to make my own toy function to experiment with (simple) recursive functions :grinning:

struct B
    x1::UInt8
    x2::UInt8
    x3::UInt8
    x4::UInt8
end

x = Tuple(B(rand(UInt8, 4)...) for _ in 1:4)

struct C
    y1::B
    y2::B
end

y=[C(x[1],x[3]), C(x[2],x[4])]  



function flat(x, f=[])
    x=!isa(x,Array) ? [x] : x
    ln=mapreduce(n -> [getfield(x[n], m) for m in 1:nfields(x[n])] , vcat, 1:length(x))
    for e in ln
        eltype(e) <: Real ? push!(f,e) : flat(e,f)
    end
    f
end


flat(y)

struct Doo{X,Y,Z}
    z1::X
    z2::Y
    z3::Z
end

nested = Doo(Doo(1, 2, 3), 4.0, 5.0)

flat(nested)

using  Flatten


flat(nested)
[flatten(nested)...]

 flat(y) # 6-element Vector{Any}:  0x6,  0xc9, ...
 
[flatten(y,Int)...] # Any[]
27

This seems to work better, but I don’t know how general it is

struct B1
    x1::Rational
    x2::Rational
    x3::Rational
    x4::Rational
end


x = Tuple([B1([Rational(rand(-20:20,2)...) for _ in 1:4]...) for _ in 1:4])

struct C1
    y1::B1
    y2::B1
end

function flatt(x, f=[])
    m=x-> hasmethod(iterate, Tuple{typeof(x)}) ? x :  [getfield(x, i) for i in 1:nfields(x)]
    for e in m(x)
        eltype(e) <: Real && length(e)==1 ? push!(f,e) : flatt(e,f)
    end
    f
end

y=[C1(x[1],x[3]), C1(x[2],x[4])]
y2=[[2//3,y], [7//8]]

flatt(y2)


y3=[[[2//3,y], 7//8, (1.0, [1,2,(3,4)])]]

flatt(y3)



y4=[[[2//3,y], 7//8, 'a', (1.0, [1,2,(3,4, -2+1.0*im)]),"abc",[3+2*im]]]




function flatt1(x, f=[])
    m=x-> hasmethod(iterate, Tuple{typeof(x)}) ? x : [getfield(x, i) for i in 1:nfields(x)]
    foreach(e->isbitstype(typeof(e)) && length(e)==1 ? push!(f,e) : flatt1(e,f), m(x))
    f
end

flatt1(y4)
28

but does flatten only work for nested structures of the same type?

Flatten works with any julia objects. For example, ModelParameters.jl uses it to get parameters from any arbitrary user defined “model”.

You can do it easily without @generated, but it will be slow - the compiler wont follow types through the recursion, it will just give up. That’s why we generate the code - to force the compiler to compile it all away for us.

1 Like