Can't access large SparseMatrix using index

New to Julia
1

Hey! I am trying to access a large SparceMatrix (205440×102400) using the code below and it gives me a “BoundsError”. For small values of N = 4 and M = 4, I don’t have any problem. Could someone help me? Thanks!!

using SparseArrays

N = 320;
M = 320;
nP = N*M;
nUV = 2*N*(M+1);
D = spzeros(nUV,nP);

auxP = (1:N-1)' .+ N;
auxU = (2:N)' .+ (N+1);

D[auxU + (auxP.-1)*nUV] = ones(N-1)

Here my “versioninfo()”

Julia Version 1.0.4
Commit 38e9fb7f80 (2019-05-16 03:38 UTC)
Platform Info:
  OS: Linux (i686-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4510U CPU @ 2.00GHz
  WORD_SIZE: 32
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

And the error is the following:

BoundsError: attempt to access 205440×102400 SparseMatrixCSC{Float64,Int32} at index [[65741123 65946564 66152005 66357446 66562887 66768328 66973769 67179210 67384651 67590092 67795533 68000974 68206415 68411856 68617297 68822738 69028179 69233620 69439061 69644502 69849943 70055384 70260825 70466266 70671707 70877148 71082589 71288030 71493471 71698912 71904353 72109794 72315235 72520676 72726117 72931558 73136999 73342440 73547881 73753322 73958763 74164204 74369645 74575086 74780527 74985968 75191409 75396850 75602291 75807732 76013173 76218614 76424055 76629496 76834937 77040378 77245819 77451260 77656701 77862142 78067583 78273024 78478465 78683906 78889347 79094788 79300229 79505670 79711111 79916552 80121993 80327434 80532875 80738316 80943757 81149198 81354639 81560080 81765521 81970962 82176403 82381844 82587285 82792726 82998167 83203608 83409049 83614490 83819931 84025372 84230813 84436254 84641695 84847136 85052577 85258018 85463459 85668900 85874341 86079782 86285223 86490664 86696105 86901546 87106987 87312428 87517869 87723310 87928751 88134192 88339633 88545074 88750515 88955956 89161397 89366838 89572279 89777720 89983161 90188602 90394043 90599484 90804925 91010366 91215807 91421248 91626689 91832130 92037571 92243012 92448453 92653894 92859335 93064776 93270217 93475658 93681099 93886540 94091981 94297422 94502863 94708304 94913745 95119186 95324627 95530068 95735509 95940950 96146391 96351832 96557273 96762714 96968155 97173596 97379037 97584478 97789919 97995360 98200801 98406242 98611683 98817124 99022565 99228006 99433447 99638888 99844329 100049770 100255211 100460652 100666093 100871534 101076975 101282416 101487857 101693298 101898739 102104180 102309621 102515062 102720503 102925944 103131385 103336826 103542267 103747708 103953149 104158590 104364031 104569472 104774913 104980354 105185795 105391236 105596677 105802118 106007559 106213000 106418441 106623882 106829323 107034764 107240205 107445646 107651087 107856528 108061969 108267410 108472851 108678292 108883733 109089174 109294615 109500056 109705497 109910938 110116379 110321820 110527261 110732702 110938143 111143584 111349025 111554466 111759907 111965348 112170789 112376230 112581671 112787112 112992553 113197994 113403435 113608876 113814317 114019758 114225199 114430640 114636081 114841522 115046963 115252404 115457845 115663286 115868727 116074168 116279609 116485050 116690491 116895932 117101373 117306814 117512255 117717696 117923137 118128578 118334019 118539460 118744901 118950342 119155783 119361224 119566665 119772106 119977547 120182988 120388429 120593870 120799311 121004752 121210193 121415634 121621075 121826516 122031957 122237398 122442839 122648280 122853721 123059162 123264603 123470044 123675485 123880926 124086367 124291808 124497249 124702690 124908131 125113572 125319013 125524454 125729895 125935336 126140777 126346218 126551659 126757100 126962541 127167982 127373423 127578864 127784305 127989746 128195187 128400628 128606069 128811510 129016951 129222392 129427833 129633274 129838715 130044156 130249597 130455038 130660479 130865920 131071361]]

Stacktrace:
 [1] throw_boundserror(::SparseMatrixCSC{Float64,Int32}, ::Tuple{Array{Int32,2}}) at ./abstractarray.jl:484
 [2] checkbounds at ./abstractarray.jl:449 [inlined]
 [3] _setindex! at ./multidimensional.jl:638 [inlined]
 [4] setindex!(::SparseMatrixCSC{Float64,Int32}, ::Array{Float64,1}, ::Array{Int32,2}) at ./abstractarray.jl:998
 [5] top-level scope at In[265]:11
2

It works for me. I’m guessing that the problem is that you are running a 32-bit Julia, and the index computations are overflowing 32 bits since length(D) is greater than typemax(Int32).

One option would be to switch to a 64-bit Julia, which is probably a good general idea if you want to work with such large matrices.

Another may be to not use “linear” (column-major) index calculations, which are awkward and probably suboptimal anyway for sparse matrices. Instead of computing linear indices directly, you could use an array of CartesianIndex:

using SparseArrays

N = 320;
M = 320;
nP = N*M;
nUV = 2*N*(M+1);
D = spzeros(nUV,nP);

indices = CartesianIndex.((2:N) .+ (N+1), (1:N-1) .+ N)
D[indices] .= 1

That being said, this is quite an inefficient way to construct sparse arrays — inserting nonzero elements is expensive. If you can, it’s much better to do something like:

D = sparse((2:N) .+ (N+1), (1:N-1) .+ N, ones(N-1), nUV,nP)

PS. Julia 1.0.4 is quite old; I would strongly recommend upgrading.

4 Likes
3

Thanks very much, that has my problem! I’ve changed to 64bit Julia version and now it works :slight_smile: I will try to use CartesianIndex , thanks for this tip too.