# Fast read operations from a large array of low-dimensional SVectors

**URL:** <https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264>\
**Category:** New to Julia\
**Tags:** performance, speed-optimization\
**Created:** [July 20, 2024, 4:56am UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264 "2024-07-20T04:56:03Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dchatter](https://avatars.discourse-cdn.com/v4/letter/d/ecae2f/32.png) [@dchatter](https://discourse.julialang.org/u/dchatter)\
**Post date:** [July 20, 2024, 4:56am UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/1 "2024-07-20T04:56:03Z")

</div>

I am looking for a suitable data structure to store the graph of a real-valued function on a rectangular grid in d dimensions; the purpose is given below. The dimension d is at most 5, and SVectors are quite suitable. However, the number of grid points could be large (tens of thousands through hundreds of thousands).

The purpose is to read the values of the function at the grid points, multiply with entries from some other (fixed) array, and sum to produce a number; all this must be done as fast as possible. Both the grid and the values at the grid points are immutable.

What would be the most appropriate data structure for the grid and the data on the grid?

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [July 20, 2024, 5:53am UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/2 "2024-07-20T05:53:01Z")

</div>

Read from RAM or from a file?

I would have a look at [GitHub - apache/arrow-julia: Official Julia implementation of Apache Arrow](https://github.com/apache/arrow-julia) . You can store the vectors in an arrow table. But you cannot avoid copying if you use SVectors.

---

<div class="post-metadata">

**Author:** ![dchatter](https://avatars.discourse-cdn.com/v4/letter/d/ecae2f/32.png) [@dchatter](https://discourse.julialang.org/u/dchatter)\
**Post date:** [July 20, 2024, 6:39am UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/3 "2024-07-20T06:39:20Z")

</div>

Thank you; I missed mentioning that the read operations are from RAM.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [July 20, 2024, 8:09am UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/4 "2024-07-20T08:09:19Z")

</div>

Why not creating a `Vector` of SVectors or MVectors? Should be trivial and fast if they all have the same size. Or an n-dimensional array?

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [July 20, 2024, 1:13pm UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/5 "2024-07-20T13:13:01Z")

</div>

Definitely agree, an `Array` (of dimension appropriate for the grid) of `SVector`s (it doesn’t seem like `MVector` is appropriate) is the most obvious answer.

---

<div class="post-metadata">

**Author:** ![dchatter](https://avatars.discourse-cdn.com/v4/letter/d/ecae2f/32.png) [@dchatter](https://discourse.julialang.org/u/dchatter)\
**Post date:** [July 20, 2024, 1:52pm UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/6 "2024-07-20T13:52:09Z")

</div>

Thank you for the suggestions. I was wondering about a `Tuple` of `SVector`s, but I did a little experiment to find that a `Vector` of `SVector`s is faster although I don’t know why.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [July 20, 2024, 2:39pm UTC](https://discourse.julialang.org/t/fast-read-operations-from-a-large-array-of-low-dimensional-svectors/117264/7 "2024-07-20T14:39:18Z")

</div>

Large tuples are not efficient. They are intended for short sequences of data with static size (ie. size known at compile time).