# Very sparse matrix but still out-of-memory?

**URL:** <https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766>\
**Category:** General Usage\
**Created:** [June 18, 2018, 8:59pm UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766 "2018-06-18T20:59:32Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![MaximilianJHuber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maximilianjhuber/32/2579_2.png) [@MaximilianJHuber](https://discourse.julialang.org/u/MaximilianJHuber)\
**Post date:** [June 18, 2018, 8:59pm UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/1 "2018-06-18T20:59:32Z")

</div>

On Julia 0.6.3, a very sparse matrix with 1000 rows and 100000 columns with only 10 entries is multiplied by a vector **column-wise** :

```julia
sparse(rand(1:1000, 10), rand(1:100000, 10), rand(10), 1000, 100000) .* rand(1000, 1)

```

This throws:

> OutOfMemoryError()  
> Stacktrace:  
> [1] \_allocres at .\sparse\higherorderfns.jl:168 [inlined]  
> [2] \_diffshape\_broadcast(::Base.#_, ::SparseMatrixCSC{Float64,Int64}, ::SparseMatrixCSC{Float64,Int64}) at .\sparse\higherorderfns.jl:132  
> [3] broadcast(::Base.#_, ::SparseMatrixCSC{Float64,Int64}, ::SparseMatrixCSC{Float64,Int64}) at .\sparse\higherorderfns.jl:121

I am surprised by this, since the resulting sparse matrix will also have only 10 non-zeros. Am I missing something?

---

<div class="post-metadata">

**Author:** ![fredrikekre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fredrikekre/32/1688_2.png) [@fredrikekre](https://discourse.julialang.org/u/fredrikekre)\
**Post date:** [June 18, 2018, 9:11pm UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/2 "2018-06-18T21:11:45Z")

</div>

Did you check with a smaller example that the result is indeed a `SparseMatrixCSC` and not a `Matrix`?

---

<div class="post-metadata">

**Author:** ![MaximilianJHuber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maximilianjhuber/32/2579_2.png) [@MaximilianJHuber](https://discourse.julialang.org/u/MaximilianJHuber)\
**Post date:** [June 18, 2018, 9:15pm UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/3 "2018-06-18T21:15:50Z")

</div>

Yes:

```julia
@time sparse(rand(1:100, 10), rand(1:10000, 10), rand(10), 100, 10000) .* rand(100, 1)

```

> 0.004960 seconds (117 allocations: 15.500 MiB, 61.64% gc time)
> 
> 100×10000 SparseMatrixCSC{Float64,Int64} with 10 stored entries:  
> [29 , 870] = 0.0375947  
> [98 , 962] = 0.0684546  
> [21 , 2043] = 0.00327209  
> [57 , 2677] = 0.280373  
> [39 , 5550] = 0.12388  
> [22 , 5820] = 0.132531  
> [6 , 6948] = 0.0959543  
> [94 , 7515] = 0.486234  
> [23 , 7848] = 0.0197313  
> [64 , 8926] = 0.148429

---

<div class="post-metadata">

**Author:** ![anon94023334](https://avatars.discourse-cdn.com/v4/letter/a/e274bd/32.png) [@anon94023334](https://discourse.julialang.org/u/anon94023334)\
**Post date:** [June 19, 2018, 5:38am UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/4 "2018-06-19T05:38:02Z")

</div>

> [@MaximilianJHuber](#):
>
> sparse(rand(1:1000, 10), rand(1:100000, 10), rand(10), 1000, 100000) .\* rand(1000, 1)

I can’t reproduce the error:

```julia
julia> z = sparse(rand(1:1000, 10), rand(1:100000, 10), rand(10), 1000, 100000) .* rand(1000, 1)
1000×100000 SparseMatrixCSC{Float64,Int64} with 10 stored entries:
  [981 , 19202] = 0.0309323
  [188 , 23036] = 0.619155
  [13 , 49876] = 0.0645187
  [675 , 53900] = 0.127399
  [672 , 58168] = 0.760207
  [865 , 59250] = 0.313583
  [437 , 60015] = 0.0336611
  [454 , 62628] = 0.259509
  [650 , 63567] = 0.118465
  [91 , 75675] = 0.0690428

julia> whos()
                          Base Module
                          Core Module
                          Main Module
                           ans 781 KB 1000×100000 SparseMatrixCSC{Float…
                             z 781 KB 1000×100000 SparseMatrixCSC{Float…

```

```julia
julia> versioninfo()
Julia Version 0.6.3
Commit d55cadc350 (2018-05-28 20:20 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin17.5.0)
  CPU: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

```

Looks like you’re using Windows, though, so perhaps that’s the difference here.

---

<div class="post-metadata">

**Author:** ![bernhard](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bernhard/32/2619_2.png) [@bernhard](https://discourse.julialang.org/u/bernhard)\
**Post date:** [June 19, 2018, 6:48am UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/5 "2018-06-19T06:48:03Z")

</div>

this is working fine for me on Windows and Julia 0.6.3

---

<div class="post-metadata">

**Author:** ![tkoolen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkoolen/32/1603_2.png) [@tkoolen](https://discourse.julialang.org/u/tkoolen)\
**Post date:** [June 19, 2018, 7:49am UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/6 "2018-06-19T07:49:08Z")

</div>

On 0.6.3, I don’t run out of memory, but memory consumption definitely seems way too high:

```julia
julia> using BenchmarkTools

julia> @btime sparse(rand(1:1000, 10), rand(1:100000, 10), rand(10), 1000, 100000) .* rand(1000, 1);
  272.095 ms (32 allocations: 1.49 GiB)

```

The allocations happen during the broadcast call. I’m getting the same behavior on latest Julia nightly.

Edit: it appears that `maxnnzC` is calculated to be `100000000` in `_diffshape_broadcast`. This seems like a bit of an overestimate. You should probably open an issue.

---

<div class="post-metadata">

**Author:** ![MaximilianJHuber](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maximilianjhuber/32/2579_2.png) [@MaximilianJHuber](https://discourse.julialang.org/u/MaximilianJHuber)\
**Post date:** [June 19, 2018, 11:57am UTC](https://discourse.julialang.org/t/very-sparse-matrix-but-still-out-of-memory/11766/7 "2018-06-19T11:57:38Z")

</div>

Thank you, [done](https://github.com/JuliaLang/julia/issues/27653)!

Maybe you have more RAM than me… Increasing the number of columns will at some point throw the error, though.