Can NamedDims.jl make use of sparse arrays?

Can NamedDims.jl make use of sparse arrays?

Note that the original poster on Slack cannot see your response here on Discourse. Consider transcribing the appropriate answer back to Slack, or pinging the poster here on Discourse so they can follow this thread.
(Original message :slack:) (More Info)

Michael Abbott 2 hours ago

Sure, NamedDimsArray(sprand(5,10,0.2), (:x, :y)) * NamedDimsArray(sprand(10,0.2), :y) seems to work. Functions specific to sparse arrays may not work, e.g. nnz(ans) . It would probably be trivial to make them work (likewise make sparse-array’s printing work). Perhaps open an issue, ideally with a list of what you wish would work. (edited)

Heetbeet 2 hours ago

Cool, thanks that is good to know. I’m probably going to start using it in 3/4 months time, so only then I will know what type of problems I am going to run in to.

Michael Abbott 2 hours ago

Cool. Would be good to support, but I guess nobody else needed it yet.

Heetbeet 2 hours ago

@mcabbott would you say that these type of calculations would be easily achievable with NamedDimsArrays + SparseArrays?
https://discourse.julialang.org/t/are-there-any-libraries-for-discrete-probability-tables-in-order-to-do-these-type/54990I’m looking to put quite some effort into implementing it, so it would be cool to know if I’m looking at the right stuff

Yesterday at 3:17 PM

Michael Abbott 2 hours ago

Maybe. I think the built-in sparse arrays are only vectors & matrices, so if you need things with 3 indices, then maybe you want GitHub - Jutho/SparseArrayKit.jl: Sparse multidimensional arrays using DOK or COO format, with support for TensorOperations.jl instead? That ought to work with TensorOperations to perform arbitrary contractions.

Michael Abbott 2 hours ago

NamedDims.jl is completely N-dimensional. My (slightly rickety!) extension in NamedPlus.jl has some things to do contractions on a given name, which calls TensorOperations, and thus might work well with its SparseArrays. But I haven’t tried that out, you’d have to explore a bit.

Heetbeet 2 hours ago

Ah okay, so the inbuild sparse arrays can’t handle n-dimensions. Good to be aware of.

Heetbeet 2 hours ago

And another implementation thing that just poped into my head, would something like swapaxes preserve the sparseness?

Heetbeet 2 hours ago

Cool, my research list is just getting longer and longer (but in a good way). TODO: check NamedPlus.jl operations (edited)

Michael Abbott 1 hour ago

Yes, I think there are many tricks for multiplying sparse matrices, etc, but for N-dims you fall back to simpler algorithms. Not sure but I presume these would implement a method for permutedims which preserves sparseness.

Heetbeet 1 hour ago

Okay. I take it the underlying data structure would be something like a map of indices to values. Which means it wouldn’t be too difficult to implement axis reordering, by just reordering the columns of the mapping (if I’m correct about the underlying implementation).

Michael Abbott 1 hour ago

Yes, I think it’s just a dictionary.

Heetbeet 1 hour ago

I think the last time we created this in C++ we unknowingly just implemented most of features of a n dimensional sparse array without realising it. All wrapped in very “DiscreteTable” specific interface. What a waste of effort!

Heetbeet 1 hour ago

Oh, another thing I just realised, is there a way to set a value other than “0” as the sparse value? Our implementation handles that. If you dot multiply A of default val 3 with B of default val 5, the resulting default val would be 15

Heetbeet 1 hour ago

This comes in handy if you set the “I don’t know” values of a discrete table as 0.5 (max-normalised). Then those values can be kept sparse and they propagate nicely through the whole sistem.

Michael Abbott 1 hour ago

That I don’t know, I didn’t think the built-in ones did this. Would not be hard to add to dict-like ones, I presume.

Related: Are there any libraries for discrete probability tables in order to do these type - #2 by heetbeet

1 Like