# Performant table representation

**URL:** <https://discourse.julialang.org/t/performant-table-representation/129880>\
**Category:** New to Julia\
**Tags:** tables\
**Created:** [June 14, 2025, 2:18pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880 "2025-06-14T14:18:21Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gesee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gesee/32/217235_2.png) [@Gesee](https://discourse.julialang.org/u/Gesee)\
**Post date:** [June 14, 2025, 2:18pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/1 "2025-06-14T14:18:21Z")

</div>

Actually I am trying to represent a table in Julia. Each column has a name and we should be able to add new column on the fly.  
Getting column and row should be fast since these are frequent operations on my work.  
Is there any package or tool matching these characteristics ?

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [June 14, 2025, 2:37pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/2 "2025-06-14T14:37:19Z")

</div>

Did you check whether the standard DataFrames.jl are good enough for your purposes? I think they are quite okay performancewise. I never did any benchmarks per se but for data analysis of O(100k) rows there was not much delay.  
There is quite a bit ecosystem around these, so if they work for you chances are that you find packages for all sorts of operations with them.

---

<div class="post-metadata">

**Author:** ![Gesee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gesee/32/217235_2.png) [@Gesee](https://discourse.julialang.org/u/Gesee)\
**Post date:** [June 14, 2025, 4:04pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/3 "2025-06-14T16:04:08Z")

</div>

Yeah, I checked and it’s a bit too slow, since I am on a real time project

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 14, 2025, 4:46pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/4 "2025-06-14T16:46:06Z")

</div>

In Julia, you don’t generally need fancy specialized data structures to have performant tables. Some basic tables are:

- vector of namedtuples, `[(a=1, b="x"), ...]` (row-based)
- namedtuple of vectors, `(a=[1,2,3,...], b=["x", "y", ...])` (column-based)
- StructArrays.jl (column-based storage with row-based interface)

Probably, some of these will work for you 🙂

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [June 14, 2025, 5:03pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/5 "2025-06-14T17:03:52Z")

</div>

Can you provide an example (MWE)? It is not clear to me:

- how large will your table be? Which data types do you need?
- what do you want to use it for? Mainly reading, or writing, or querying?
- what are your performance requirements? Which operation must be how fast? Are no memory allocations allowed, or some?

We cannot really help you without more details and preferably a test case and test data.

---

<div class="post-metadata">

**Author:** ![Gesee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gesee/32/217235_2.png) [@Gesee](https://discourse.julialang.org/u/Gesee)\
**Post date:** [June 14, 2025, 5:27pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/6 "2025-06-14T17:27:59Z")

</div>

Actually, the table size may reach im extreme case 5B elements  
I need to perform resizing on the table frequently (adding rows or columns) and getting data from row or column.  
Since it’s a real time project, taking more than 100ns for these operations may not suit me since they are quite frequent  
Also, the table grow but don’t shrink.  
I will frequently read, write but no querying

---

<div class="post-metadata">

**Author:** ![Gesee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gesee/32/217235_2.png) [@Gesee](https://discourse.julialang.org/u/Gesee)\
**Post date:** [June 14, 2025, 5:29pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/7 "2025-06-14T17:29:53Z")

</div>

I am already exploring StructArrays.jl the problem is that in my case, I need to be able to grow the array, and we can’t change the size of a tuple nor a named tuple, it’s quite complicated

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [June 14, 2025, 5:35pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/8 "2025-06-14T17:35:42Z")

</div>

> [@Gesee](#):
>
> 5B elements

Do you mean 5 billion elements? 5e9 elements? or 5e12 elements? (You might know that billion is defined differently in US and Europe).  
And what would be the size of one element?

Again, without a concrete test case, please don’t expect many useful answers.

---

<div class="post-metadata">

**Author:** ![lmiq](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lmiq/32/18314_2.png) [@lmiq](https://discourse.julialang.org/u/lmiq)\
**Post date:** [June 14, 2025, 5:37pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/9 "2025-06-14T17:37:56Z")

</div>

Maybe a dictionary? Adding “columns” would be fast, adding rows would require pushing to each vector, but that’s quite fast, as memory movements are minimized automatically by the resizing mechanism.

(That said, I’m not sure if what DataFrames does is in the limit of what can be done performance wise)

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 14, 2025, 5:47pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/10 "2025-06-14T17:47:06Z")

</div>

> [@Gesee](#):
>
> I need to be able to grow the array, and we can’t change the size of a tuple nor a named tuple, it’s quite complicated

With (named)tuples, it’s “free” to just create a new one! Adding/removing a StructArray column doesn’t copy any data, even though it creates a new NamedTuple and StructArray.

```julia
julia> using StructArrays, Accessors

julia> tbl = StructArray(a=[1,2,3], b=[4,5,6])
3-element StructArray(::Vector{Int64}, ::Vector{Int64}) with eltype @NamedTuple{a::Int64, b::Int64}:
 (a = 1, b = 4)
 (a = 2, b = 5)
 (a = 3, b = 6)

julia> @insert tbl.c = rand(3)
3-element StructArray(::Vector{Int64}, ::Vector{Int64}, ::Vector{Float64}) with eltype @NamedTuple{a::Int64, b::Int64, c::Float64}:
 (a = 1, b = 4, c = 0.6328852033561965)
 (a = 2, b = 5, c = 0.6543847896731916)
 (a = 3, b = 6, c = 0.49184492764291854)

```

---

<div class="post-metadata">

**Author:** ![Gesee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gesee/32/217235_2.png) [@Gesee](https://discourse.julialang.org/u/Gesee)\
**Post date:** [June 14, 2025, 6:17pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/11 "2025-06-14T18:17:08Z")

</div>

I think I will use this, I wasn’t sure if having to go through the whole dictionary to reconstruct a row would be good for performances

---

<div class="post-metadata">

**Author:** ![cgarling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cgarling/32/29278_2.png) [@cgarling](https://discourse.julialang.org/u/cgarling)\
**Post date:** [August 8, 2025, 1:33pm UTC](https://discourse.julialang.org/t/performant-table-representation/129880/12 "2025-08-08T13:33:21Z")

</div>

I usually use [TypedTables.jl](https://typedtables.juliadata.org/stable/) when I want something simple and fast. StructArrays.jl is nice for more complicated use-cases.