More specific definition for DenseArray

The documentation of a DenseArray{T,N} says:

N-dimensional dense array with elements of type T.
The elements of a dense array are stored contiguously in memory.

but this is a bit unspecific. For instance, my understanding is that if the IndexStyle of a DenseArray is IndexLinear() then it can safely be assumed that successive elements of this array are stored contiguously in memory. If my assumption is true, this should be stated more clearly in the documentation. This would be useful to generalize which kinds of arrays can be safely passed to external libraries.

I think that the implicit assumption is that the user will always access the elements via the API (which includes magic like eachindex) and It Will Just Work™. Only internal methods need to know the actual layout.

EDIT the manual actually mentions that DenseArrays is column major, so I guess that the docstring could do this, too. But I am still not sure that this is a type that needs to be exposed.

Note that DenseArray is abstract. A descendant can be row-major instead of column major.

TBH, I think is an abstract type used internally for organizing the implementation, and should not have been exported at all. It was swept up in

and documented in

I don’t think a user should need to use this type with the current API, ever.

Thank you for pointing this chapter, it answers my questions exactly. I should have better searched the documentation…

Using this abstract type and knowing a bit more what it assumes (or imposes) for the storage of the array elements has some advantages for my applications (real-time control) and certainly for others:

  • DenseArray can be used to write a method which calls a numerical library (like BLAS or LAPACK but not only) in such a way that it remains applicable to other types than just Array.

  • New array types may inherit from DenseArray and benefit from the above method with no other efforts than making sure that their storage really makes them dense arrays. In that spirit I have implemented Julia wrappers around various array-like objects that are stored by other pieces of software in shared memory.

I admit that all these are rather low-level considerations, but this is what I like with Julia: its agility when one needs to fuse high level code with low level or foreign code.

This is a very valid use case, which I believe is served currently by StridedArray. Also see strides.

Again, I don’t think there is any reason for a user (ie someone writing code outside Base) to even know about DenseArray. AFAIK it is a remnant for an earlier API.

As far as I understand the sentence:

DenseArray is an abstract subtype of AbstractArray intended to include all arrays where elements are stored contiguously in column-major order.

DenseArray is a bit more specific than StriddedArray in the sense that to check that elements of a strided array, say A are stored contiguously in column-major order (like in a DenseArray) it must be asserted that the strides of A are (1, size(A,1), size(A,1)*size(A,2), ...).

The additional information summarized by DenseArray can be very useful to write shorter code and deserves being exposed in the API.

If you look at the above extract from the documenation (pointed by Tamas Papp), a DenseArray must be column-major.

In theory, yes, but in practice most Fortran code takes strided arrays (and C is, of course, row-major).

That said, I actually agree that knowing this is a valid use case, I just don’t think that <: DenseArray is the right way to expose this. The reason for this is that type hierarchies are very limiting (because of single inheritance), so the preferred API design now uses traits for these things.

Also, whenever you are interfacing with C/Fortran, you still need an explicit mapping. Should the fact that something is a DenseArray be exposed? Is it relevant to the user, once the mapping to C and the element access/iteration API is defined efficiently?

OK but this implies that the relevant trait (to be defined) be implemented for the various array types. In that particular case, DenseArray is sufficient and needs no extra work.

It can be used for this yes.
But DenseArray on its own is neither necessary
(aforemntioned StridedArray)
nor surficient: a DenseArray of BigInt is not going too use BLAS.

Traits are the appropriate way to expose this.
There is an ongoing effort in this space:

But yes, that doesn’t change that we could document DenseArray better.

Its worth openning issues about this on the GitHub repo, where such things are tracked.

1 Like

You are right DenseArray{T,N} must be specified in the signature with consistent type T and number of dimensions N.

I was not aware of the ArrayInterface effort. This looks very interesting!

For the documentation, I can post an issue but as it is a matter of adding a single sentence in the docstring a PR may be more appropriate. No?