Performance vs notation tradeoff: DataFrame vs AxisArray vs StaticArray vs?

Tetrakai
Sat, 17 Aug 2024 22:33:06 +0000
Performance

I’ve got two datasets that would look like this in a dataframe (see later for MWE):

julia> df1 = mat2df(mat1, colnames1, id1)
8×5 DataFrame
 Row │ id      col1   col2   col3   col4
     │ String  Int64  Int64  Int64  Int64
─────┼────────────────────────────────────
   1 │ a          32     49     96     63
   2 │ b          55     75     65     23
   3 │ c          22     58    100     12
   4 │ d          90     73     75     61
   5 │ e          35      0     11     67
   6 │ f          39     20     49     76
   7 │ g          96     44     57     59
   8 │ h          80     68     25     36

julia> df2 = mat2df(mat2, colnames2, id2)
4×7 DataFrame
 Row │ id      col1   col2   col3   col6   col4   col5
     │ String  Int64  Int64  Int64  Int64  Int64  Int64
─────┼──────────────────────────────────────────────────
   1 │ d           1      6      9      5      9      5
   2 │ b          10      0      3      3      7     10
   3 │ c           6      7      8      2      4      8
   4 │ a           7      1      2     10      9      5

I need to update (in various ways depending on the column) only some of the rows/columns of df1 based on the values in df2. The dataframe is so easy to look at and I love the succinct dot notation (ie, df1.col1[idx] vs df1[idx, "col1"] vs df1[idx, 2]), but it is far too slow.

So I tried out AxisArrays, and it allows me to index by name while being ~10x faster. However, there was still no dot notation.

Then I tried StaticArrays, and it was somewhat faster than an AxisArray (afaict, I still needed to use a mutable MArray here), but doesn’t allow any indexing by name. Further, it seems I need to hardcode the dimensions (I can’t pass it as a function argument?)

Ie, compare updateSA vs updateSA2 functions below. The only difference is: MMatrix{sz1[1], sz1[2]}(mat1) vs MMatrix{8, 4}(mat1). The first option is slow as a dataframe.

Here’s the MWE (I’m not worried about the copy(mat1), that only mattered for the benchmark):
MWE (click for more details)
Benchmarks:

julia> @btime updateDF(copy($mat1), $id1, $colnames1, $mat2, $id2, $colnames2);
  8.031 μs (101 allocations: 7.12 KiB)

julia> @btime updateAA(copy($mat1), $id1, $colnames1, $mat2, $id2, $colnames2);
  1.813 μs (16 allocations: 1.73 KiB)

julia> @btime updateSA(copy($mat1), $id1, $colnames1, $mat2, $id2, $colnames2);
  8.441 μs (68 allocations: 4.70 KiB)

julia> @btime updateSA2(copy($mat1), $id1, $colnames1, $mat2, $id2, $colnames2);
  1.647 μs (16 allocations: 2.17 KiB)

1. Is there a way to get index, string, and dot notation (like dataframe) with near the speed of StaticArray? Or is this tradeoff just inherent to having multiple indexing methods?

2. Is there some way to pass a variable for the dimensions of a StaticArray without slowing it down 10x?

3. Is MMatrix the right solution for the problem in the MWE?

Thanks for any help.

3 posts - 2 participants