# Obtaining field values over an array of composite types

**URL:** <https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518>\
**Category:** Performance\
**Tags:** struct\
**Created:** [October 21, 2024, 12:07am UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518 "2024-10-21T00:07:44Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![miguelborrero](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@miguelborrero](https://discourse.julialang.org/u/miguelborrero)\
**Post date:** [October 21, 2024, 12:07am UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/1 "2024-10-21T00:07:44Z")

</div>

Hi there,

I defined a composite type `Product` tho store relevant characteristics of a set of products. I then construct an array of Product and I am ultimately interested in accessing the field values of this array as efficient as possible. I have tried different ways of doing so and I am getting by far the best performance using list comprehension. I was hoping someone could give intuition as to way this is happening and/or provide with the optimal way. Thanks a lot!

```julia
using DataFrames
using BenchmarkTools
# define a product structure
struct Product{T}
    id::Int64
    X0::T
    X1::T
    X2::T
end

function initialize_products(data::DataFrame, number_products::Int64)
    # initialize product array 
    products = Array{Product}(undef, number_products)
    for i in 1:number_products
        products[i] = Product(collect(data[i, :])...)
    end
    return products 
end

function comprehension(products)
    return [product.X0 for product in products]
end

function get_fields(products)
    return getfield.(products, :X0)
end

function using_map(products)
    return map(product -> product.X0, products)
end

function main()
    df = DataFrame(product_id = [2, 3, 4, 6, 8, 9, 10],
                          X0 = ones(7),
                          X1 = rand(7),
                          X2 = rand(7),
                )
    products = initialize_products(df, 7)
    println(@btime comprehension($products))
    println(@btime get_fields($products))
    println(@btime using_map($products))
end

main()

```

the result being:  
 ![Captura de pantalla 2024-10-20 a la(s) 17.07.38](https://global.discourse-cdn.com/julialang/original/3X/e/f/ef201dc88ca82414dcd012e27ab0325739ded77f.png)

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 21, 2024, 1:48am UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/2 "2024-10-21T01:48:06Z")

</div>

> [@miguelborrero](#):
>
> `products = Array{Product}(...)`

Noticing that you manually specified an abstract type for the array’s element type, which will get in the way of inference of the `X_::T` fields. Comprehensions, broadcasting, and `map` don’t infer element types and initialize output arrays the same way, but the printouts suggest that they all did some work to infer `Vector{Float64}` outputs. The more difficult `Array{Product}` input could be exacerbating the performance discrepancy.

`T::Float64` for your example’s inputs, so you could try `Array{Product{Float64}}`. If you want to make it a bit more flexible, you can first collect the `Array{Product,1}` first, but check in the loop if all the `Product(...)` calls result in the same type as the first element’s, let’s call it `F`, in which case you can copy to a more specific `Array{F,1}`.

---

<div class="post-metadata">

**Author:** ![miguelborrero](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@miguelborrero](https://discourse.julialang.org/u/miguelborrero)\
**Post date:** [October 21, 2024, 1:56am UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/3 "2024-10-21T01:56:25Z")

</div>

thanks for the reply but Im a beginner and I don’t really understand what you say.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 21, 2024, 3:38am UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/4 "2024-10-21T03:38:39Z")

</div>

> [@miguelborrero](#):
>
> `products = Array{Product}(undef, number_products)`

> [@Benny](#):
>
> try `Array{Product{Float64}}`.

Do this bit of type parameter optimization first and show how the benchmark changes on your system.  
Another optimization you could do to get around the automatic output allocation in broadcasting and `map` is to use in-place broadcasting and `map!` to mutate an output array you allocate manually.

> [@miguelborrero](#):
>
> Im a beginner and I don’t really understand what you say.

You code very well for one! I don’t know what you need to know to understand my previous comment, but I’m mainly talking about Julia’s [type system](https://docs.julialang.org/en/v1/manual/types/), and type inference is an attempt to narrow down types of variables, elements, and expressions, often by the compiler or instantiation of containers. Poor type inference shouldn’t change runtime values, but the compiled code must spend extra time to figure out types and appropriate methods at runtime. [`@code_warntype`](https://docs.julialang.org/en/v1/stdlib/InteractiveUtils/#InteractiveUtils.@code_warntype) and similar reflection methods and macros can help you see type inference, though I’ll warn these assume the runtime types of the input call’s arguments are inferred perfectly (aka specializing), which you may [manually opt out](https://docs.julialang.org/en/v1/base/base/#Base.@nospecialize) of or the compiler intentionally does not do [in rare circumstances](https://docs.julialang.org/en/v1/manual/performance-tips/#Be-aware-of-when-Julia-avoids-specializing).

That is quite a bit of reading to do, so if you have any more targeted questions that can clear things up, feel free to ask.

---

<div class="post-metadata">

**Author:** ![arnold-c](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/arnold-c/32/222741_2.png) [@arnold-c](https://discourse.julialang.org/u/arnold-c)\
**Post date:** [October 21, 2024, 4:40am UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/5 "2024-10-21T04:40:01Z")

</div>

Aside from the advice above, you may want to look into using a structure-of-arrays instead of the array-of-structures form you have here. See [Overview · StructArrays](https://juliaarrays.github.io/StructArrays.jl/stable/) for more information, as the StructArrays.jl package has a bunch of convenient features, as well as generally being quite efficient - I’m on my phone so can’t benchmark on your code, but I’ve generally had good results (or at the very least, haven’t noticed a slowdown that would outweigh the convenience).

---

<div class="post-metadata">

**Author:** ![miguelborrero](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@miguelborrero](https://discourse.julialang.org/u/miguelborrero)\
**Post date:** [October 21, 2024, 2:51pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/6 "2024-10-21T14:51:39Z")

</div>

Wow! It really did change when I implemented first change suggested:

 ![Captura de pantalla 2024-10-21 a la(s) 07.50.51](https://global.discourse-cdn.com/julialang/original/3X/d/3/d328221592a30056ad5adf5eb5b3e29cc214c44f.png)

Thanks a lot. Do you have intuition as to way broadcasting has more allocations?

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [October 21, 2024, 3:04pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/7 "2024-10-21T15:04:04Z")

</div>

I think the compiler doesn’t “know” the field name when you do `getfield.(products, :X0)`. Here, `:X0` is a regular runtime value, which requires dispatch when actually running the code.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [October 21, 2024, 3:07pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/8 "2024-10-21T15:07:04Z")

</div>

Also, your `initialize_products` function could just be something like this:

```julia
using Tables

products = map(row -> Product(row...), rowtable(dataframe))
# or, to handle arbitrarily ordered columns
products = map(row -> Product(;row...), rowtable(dataframe))

```

Depending on how you get the original dataset, you may not need dataframes at all: just read your data into a Vector or StructArray in the first place.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [October 21, 2024, 5:50pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/9 "2024-10-21T17:50:01Z")

</div>

I don’t use DataFrames, so running your MWE is a bit difficult. And, really, DataFrames is totally irrelevant to the question, so I suggest you initialize your data like this

```julia
products = [Product(i, 1.0, rand(), rand()) for i in [2, 3, 4, 6, 8, 10]]

```

As for broadcasting, you can make it equally fast as the other methods, just remember to move the broadcasting to the outermost level:

```julia
getX0(p) = getfield(p, :X0)

julia> @btime comprehension($products);
  46.802 ns (1 allocation: 112 bytes)

julia> @btime getX0.($products);
  42.698 ns (1 allocation: 112 bytes)

```

(Everything is a bit slow here, running on battery.)

---

<div class="post-metadata">

**Author:** ![miguelborrero](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@miguelborrero](https://discourse.julialang.org/u/miguelborrero)\
**Post date:** [October 21, 2024, 5:55pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/10 "2024-10-21T17:55:47Z")

</div>

Oh wow, interesting. In terms of dataframes I am constrained by the application but you are right, this is irrelevant. In terms of the broadcasting: do you know why this solution works? Thanks a lot!

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [October 21, 2024, 6:22pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/11 "2024-10-21T18:22:16Z")

</div>

I’m not totally sure, but it might perhaps be more difficult to constant propagate through the broadcasting call, while `getfield(p, :X0)` is sure to be optimized.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [October 21, 2024, 8:32pm UTC](https://discourse.julialang.org/t/obtaining-field-values-over-an-array-of-composite-types/121518/12 "2024-10-21T20:32:50Z")

</div>

> [@miguelborrero](#):
>
> In terms of the broadcasting: do you know why this solution works?

To elaborate, with the exception of a few types (`Function`, `Type`, `Vararg`), Julia’s compiler specializes method calls on the argument types by default. That means if you have a very flexible method `function foo(x::Any)`, and you call `foo(1)`, the compiler will compile that `foo` method specifically for an `Int` input. Narrowing down types throughout the method is important for efficiency.

Thing is, that’s not good enough for something like `getfield(p, :X0)`. The runtime type of `:X0` is `Symbol`, and that doesn’t narrow down any field or its type. So, the compiler effectively specializes on the constant _value_ of `:X01` when it sees the `getfield(p, :X0)` call in another method it’s compiling, like `getX0`. BenchmarkTools actually wraps the benchmarked code in a method so there is compilation going on, but it doesn’t help `getfield.(products, :X0)` because it is transformed to an indirect `broadcast(getfield, products, :X0)`, the `getfield(p, :X0)` call is gone. It’s not impossible, but it’s more complicated to say that `broadcast` or whatever related methods will specialize on the constant value of the 3rd argument if the 1st argument is `getfield`, and the rule is too indirect to identify and work on different methods that internally call `getfield`.

Thankfully it’s not too hard to put the `:X0` into a direct `getfield` call for broadcasting. As an alternative to defining a named function `getX0`, you could inline an anonymous one: `(p -> getfield(p, :X0)).(products)` or the sugar syntax `(p -> p.X0).(products)`, which technically calls `getproperty` which may fall back to `getfield`.