Can NamedDims.jl make use of sparse arrays?
Note that the original poster on Slack cannot see your response here on Discourse. Consider transcribing the appropriate answer back to Slack, or pinging the poster here on Discourse so they can follow this thread.
(Original message 
) (More Info)
Sure, NamedDimsArray(sprand(5,10,0.2), (:x, :y)) * NamedDimsArray(sprand(10,0.2), :y) seems to work. Functions specific to sparse arrays may not work, e.g. nnz(ans) . It would probably be trivial to make them work (likewise make sparse-array’s printing work). Perhaps open an issue, ideally with a list of what you wish would work. (edited)
Cool, thanks that is good to know. I’m probably going to start using it in 3/4 months time, so only then I will know what type of problems I am going to run in to.
Cool. Would be good to support, but I guess nobody else needed it yet.
@mcabbott would you say that these type of calculations would be easily achievable with NamedDimsArrays + SparseArrays?
https://discourse.julialang.org/t/are-there-any-libraries-for-discrete-probability-tables-in-order-to-do-these-type/54990I’m looking to put quite some effort into implementing it, so it would be cool to know if I’m looking at the right stuff
Yesterday at 3:17 PM
Maybe. I think the built-in sparse arrays are only vectors & matrices, so if you need things with 3 indices, then maybe you want GitHub - Jutho/SparseArrayKit.jl: Sparse multidimensional arrays using a DOK format, with support for TensorOperations.jl instead? That ought to work with TensorOperations to perform arbitrary contractions.
NamedDims.jl is completely N-dimensional. My (slightly rickety!) extension in NamedPlus.jl has some things to do contractions on a given name, which calls TensorOperations, and thus might work well with its SparseArrays. But I haven’t tried that out, you’d have to explore a bit.
Ah okay, so the inbuild sparse arrays can’t handle n-dimensions. Good to be aware of.
And another implementation thing that just poped into my head, would something like swapaxes preserve the sparseness?
Cool, my research list is just getting longer and longer (but in a good way). TODO: check NamedPlus.jl operations (edited)
Yes, I think there are many tricks for multiplying sparse matrices, etc, but for N-dims you fall back to simpler algorithms. Not sure but I presume these would implement a method for permutedims which preserves sparseness.
Okay. I take it the underlying data structure would be something like a map of indices to values. Which means it wouldn’t be too difficult to implement axis reordering, by just reordering the columns of the mapping (if I’m correct about the underlying implementation).
Yes, I think it’s just a dictionary.
I think the last time we created this in C++ we unknowingly just implemented most of features of a n dimensional sparse array without realising it. All wrapped in very “DiscreteTable” specific interface. What a waste of effort!
Oh, another thing I just realised, is there a way to set a value other than “0” as the sparse value? Our implementation handles that. If you dot multiply A of default val 3 with B of default val 5, the resulting default val would be 15
This comes in handy if you set the “I don’t know” values of a discrete table as 0.5 (max-normalised). Then those values can be kept sparse and they propagate nicely through the whole sistem.
That I don’t know, I didn’t think the built-in ones did this. Would not be hard to add to dict-like ones, I presume.