Parsimonious way to unpack big structs

General Usage
1

I often work with large structs w/ many (over 20) parameters.
I currently do something like:

struct m; α; β; γ; end  # store parameters in a struct
m(; α=0.1,β=0.2,γ=0.3) = m(α,β,γ) # fcn that returns struct w/ defaults

p = m() # e.g. store the param in `p`

# Example 1: no unpacking
f1(x; p=p) = x^(p.α) + x^(p.β) + x^(p.γ)

# Example 2: unpack w/ current version of Julia
function f2(x;p=p)
    α, β, γ = p.α, p.β, p.γ
    return x^(α) + x^(β) + x^(γ)
end

# Example 3: unpack w/ Julia 1.7+
function f3(x;p=p)
    (; α, β, γ) = p
    return x^(α) + x^(β) + x^(γ)
end

Is there a way to unpack the object p inside a function w/o repeating it’s components (; α, β, γ) = p?
I realize this sounds lazy, but when you have 20+ params in a struct & have to do (; α, β, γ) = p over many times…

I’d like something like (maybe a macro @unpack_$)

function f4(x;p=p)
    @unpack_$ p
    return x^(α) + x^(β) + x^(γ)
end

Weird shot in the dark. Maybe something like

fieldnames(typeof(p)) # returns tuple (:α, :β, :γ)
# Then have `@unpack_$ p` do something like: 
fieldnames(typeof(p)) = p.α, p.β, p.γ
1 Like
2

Well, there is the “somewhat dangerous” unpacking available from Parameters.jl. From the documentation,

function fn(var, pa::Para)
    @unpack_Para pa # the macro is constructed during the @with_kw
                    # and called @unpack_*
    out = var + a + b
end

where a and b are fields unpacked from Para. They admonish to be careful with this, as it can collide/shadow other variables.

5 Likes
3

I was just about to reply with the same reference. I wish this feature could be reworked so that it is safer to use because I often need to extract many parameters from a struct.

1 Like
4

If the parameters are related in some way it might be better to group some of them together into a few named tuples instead of 30 free-floating parameters.

3 Likes
5

@apo383

  1. Happy Birthday!
  2. it works, but only w/ @with_kw struct 
julia> struct m; α; β; γ; end
julia> m(;α=0.1,β=0.2,γ=0.3) = m(α,β,γ);
julia> p = m();

julia> f1(x; p=p) = x^(p.α) + x^(p.β) + x^(p.γ);
julia> function f2(x;p=p)
           α, β, γ = p.α, p.β, p.γ
           return x^(α) + x^(β) + x^(γ)
       end;

julia> using Parameters;
julia> @with_kw struct M; α; β; γ; end;
julia> M(;α=0.1,β=0.2,γ=0.3) = M(α,β,γ);
julia> p=M();
julia> function f4(x; p::M=p)
           @unpack_M p
           return x^(α) + x^(β) + x^(γ)
       end;

julia> f1.(1:4)== f2.(1:4) == f4.(1:4)
true

If anyone has, other (better) ways, I’m all ears

6

Yes, I guess @with_kw lets the macro know about your definition. However, it might be worth it, since it allows the default parameter values to be defined at the same time, saving the need to define the extra constructor.

@with_kw struct M; α=1; β=1; γ=1; end; # no need for separate default constructor

I agree with @mauro3 to be cautious about this kind of unpacking. This was the much-derided with of Pascal (yes I am old, thanks), and also an anti-pattern in Javascript.

1 Like
7

I often find myself in this situation. I use ComponentArrays.jl and UnPack.jl instead. Is there any advantage of this approach vs the one using structures? What’s best practice?

1 Like
8

@amrods Can you show us what that looks like?

9

using your example, it would be

using ComponentArrays
using UnPack

p = ComponentArray(α=0.1, β=0.2, γ=0.3)

function f2(x; p=p)
    @unpack α, β, γ = p
    return x^(α) + x^(β) + x^(γ)
end

I also like to organize the parameters inside the ComponentArray, for example:

using ComponentArrays
using UnPack

# full model
function model(p; data=data)
    @unpack p1, p2 = p
    @unpack data1, data2 = data
    
    # component 1
    r1 = f1(p1; data=data1)

    # component 2
    r2 = f2(p2; data=data2)

    return sum(r1 .* r2)
end

function f1(p1; data=data)
    @unpack α, β = p1
    @unpack x, y = data
    α .* x + β .* y
end

function f2(p2; data=data)
    @unpack γ, ψ = p2
    @unpack x, y = data
    γ[1] .* x[:, 1] + γ[2] .* x[:, 2] + ψ .* y
end

data = ComponentArray(data1 = (x = rand(100), y = rand(100)),
                      data2 = (x = rand(100, 2), y = rand(100)))
p = ComponentArray(p1 = (α = 1.0, β = 0.5),
                   p2 = (γ = [10.0, 15.0], ψ = 11.0))

model(p; data=data)
2 Likes
10

I should also say that you can then throw model into an optimizer:

opt = optimize(p -> -model(p; data=data), p)

That’s the advantage of ComponentArrays.jl.

4 Likes
11

Just keep it explicit… This is my favourite version:

function f2(x;p=p)
    α, β, γ = p.α, p.β, p.γ
    return x^(α) + x^(β) + x^(γ)
end;

No one will ever be confused, and confusion wastes much more time than typing. It also means you can use longer names in your struct and symbols in the actual math, which is self-documenting.

7 Likes
13

I dont know, I think new people could be confused by @unpack, and they’'re the ones we dont want to confuse.

Early on I put macros like that everywhere in my code, but when it came to onboarding new people to shared codebase in an organisation it seemed much better to remove them all again - it made things easier to understand if we just used base julia. Especially in the context of code and models that were complicated enough already.

3 Likes
14

Edit: not sure why I deleted the last post, but here it was:

The unpacking idiom is sufficiently common that people shouldn’t be confused. @unpack α, β, γ = p is all over the place, and so will (; α, β, γ) = p before too long. But the problem with your code is one of defensive programming: α, β, γ = p.α, p.β, p.γ is extremely error prone. If you accidentally get them out of order, maybe after adding a new parameter or removing one, then you will get all sorts of silent bugs. If you have to copy/paste these between funcitons then the probability for a mistake increases further.

Users have to learn some new idioms any time to learn a new language. This is one to learn as early as possible because the alternatives have been so error prone for everyone. But in general, I agree. Other macros I am very suspicious of for new users.

And this isn’t even a macro anymore: (; a, b) = p. They will have to learn about named tuples pretty quickly, and this shiould be a part of that training.

6 Likes
15

If other people will look at the code, I’d prefer to at least give a hint with

(; a, b) = p # unpacking fields

It is cute syntactic sugar, but not very readable for the uninitiated. With @unpack at least it’s clear and someone can look up the documentation easily.

For me the beauty of Julia is that it is almost as readable as pseudo-code. I sometimes brag about the readability, only for someone to glance at it and get tripped up on something not readable. Examples include short-circuit && instead of if, and array undef.

3 Likes
16

The named tuple idiom is better as it’s in base and mirrors how named tuples splat to keyword arguments. But I guess there is some taste involved here too!

I specifically commented as I’m just about to publish a paper with similar unpacking code in inline examples. There is already too much to explain for non-julia users (symbols, type parameters, do blocks, etc…), and although it’s simple, an @unpack macro does add one more thing to the list, including what a macro is at all.

1 Like
17

That’s precisely what drove me to ComponentArrays. It is also very tedious to change the code to try different model pieces that take subsets of a large vector of parameters.

That is what I do, just with @unpack.

4 Likes
18

If I was to call this function f2a lot of times, would this version be more performant than an @unpack solution?

19

No, @unpack is a macro and expands essentially to the same code. Have checked for SimpleUnPack.@unpack and Parameters.@unpack and all of these functions reduce to the same code, i.e., at the level of @code_typed:

function unpack_1(p)
    Parameters.@unpack α, β, γ = p
    α + β + γ
end
function unpack_2(p)
    SimpleUnPack.@unpack α, β, γ = p
    α + β + γ
end
function unpack_3(p)
    α, β, γ = p.α, p.β, p.γ
    α + β + γ
end

In any case, you can always check fo yourself:

julia> @code_typed unpack_1(p)
CodeInfo(
1 ─ %1 = Base.getfield(p, :α)::Any
│   %2 = Base.getfield(p, :β)::Any
│   %3 = Base.getfield(p, :γ)::Any
│   %4 = (%1 + %2 + %3)::Any
└──      return %4
) => Any

julia> @macroexpand SimpleUnPack.@unpack α, β, γ = p
:((; α, β, γ) = p)
1 Like
20

Thanks a ton! I like that approach, thanks for showing me.

I ended up simply just doing it in my particular code and it didn’t seem to have an impact so I stuck to using @unpack since it made syntax much cleaner

Kind regards