Fast read operations from a large array of low-dimensional SVectors

New to Julia
,
1

I am looking for a suitable data structure to store the graph of a real-valued function on a rectangular grid in d dimensions; the purpose is given below. The dimension d is at most 5, and SVectors are quite suitable. However, the number of grid points could be large (tens of thousands through hundreds of thousands).

The purpose is to read the values of the function at the grid points, multiply with entries from some other (fixed) array, and sum to produce a number; all this must be done as fast as possible. Both the grid and the values at the grid points are immutable.

What would be the most appropriate data structure for the grid and the data on the grid?

2

Read from RAM or from a file?

I would have a look at GitHub - apache/arrow-julia: Official Julia implementation of Apache Arrow . You can store the vectors in an arrow table. But you cannot avoid copying if you use SVectors.

3

Thank you; I missed mentioning that the read operations are from RAM.

4

Why not creating a Vector of SVectors or MVectors? Should be trivial and fast if they all have the same size. Or an n-dimensional array?

1 Like
5

Definitely agree, an Array (of dimension appropriate for the grid) of SVectors (it doesn’t seem like MVector is appropriate) is the most obvious answer.

1 Like
6

Thank you for the suggestions. I was wondering about a Tuple of SVectors, but I did a little experiment to find that a Vector of SVectors is faster although I don’t know why.

7

Large tuples are not efficient. They are intended for short sequences of data with static size (ie. size known at compile time).