# SparseArray types over distributed reduction

**URL:** <https://discourse.julialang.org/t/sparsearray-types-over-distributed-reduction/53116>\
**Category:** Performance\
**Tags:** parallel, sparse\
**Created:** [January 10, 2021, 2:59pm UTC](https://discourse.julialang.org/t/sparsearray-types-over-distributed-reduction/53116 "2021-01-10T14:59:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tblz](https://avatars.discourse-cdn.com/v4/letter/t/eada6e/32.png) [@tblz](https://discourse.julialang.org/u/tblz)\
**Post date:** [January 10, 2021, 2:59pm UTC](https://discourse.julialang.org/t/sparsearray-types-over-distributed-reduction/53116/1 "2021-01-10T14:59:18Z")

</div>

I have a use-case of large sparse arrays that can be built in parallel. A minimum working example similar to my code is:

```julia
using Distributed

@everywhere using SparseArrays

@everywhere function build_array()
	return sparsevec(Int32.(1:10), Int32(1), Int32(10))
end

function main()
	full_array += @distributed (+) for i in 1:10
		build_array()
	end
end

main()

```

I’m trying to reduce RAM usage but noticing that summing two Int32 sparse arrays seems to lead to automatic conversion to Int64 counts. Just looking at REPL:

```julia
julia> one = sparsevec(Int32.(1:10), Int32(1), Int32(10))
10-element SparseVector{Int32,Int32} with 10 stored entries

```

compared to a sum:

```julia
julia> two = sparsevec(Int32.(1:10), Int32(1), Int32(10)) + sparsevec(Int32.(1:10), Int32(1), Int32(10))
10-element SparseVector{Int32,Int64} with 10 stored entries

```

Where you can see that the counting part of the SparseArray is now Int64 and `varinfo()` shows a correspondingly higher memory use than an identical sparse(Int32, Int32) object.

```julia
julia> equiv = sparsevec(Int32.(1:10), Int32(2), Int32(10))
10-element SparseVector{Int32,Int32} with 10 stored entries

julia> varinfo(r"one")
  name size summary                             
  –––– ––––––––– ––––––––––––––––––––––––––––––––––––
  one 184 bytes 10-element SparseVector{Int32,Int32}

julia> varinfo(r"two")
  name size summary                             
  –––– ––––––––– ––––––––––––––––––––––––––––––––––––
  two 224 bytes 10-element SparseVector{Int32,Int64}

julia> varinfo(r"equiv")
  name size summary                             
  ––––– ––––––––– ––––––––––––––––––––––––––––––––––––
  equiv 184 bytes 10-element SparseVector{Int32,Int32}

```

I understand that in principle summing many Int32’s can lead to an Int64. However in my use case I can guarantee that the sum will stay \<2 billion. I’m a bit new to using distributed workflows and sparse arrays and not quite sure what part of code needs to be improved (can the distributed reduction type be pre-specified? can SparseArray types be maintained? it doesn’t seem clear from documentation).

But would appreciate a way to maintain my construction in parallel while staying with Int32 types if that is possible. Thanks!

---

<div class="post-metadata">

**Author:** ![tblz](https://avatars.discourse-cdn.com/v4/letter/t/eada6e/32.png) [@tblz](https://discourse.julialang.org/u/tblz)\
**Post date:** [January 13, 2021, 10:20pm UTC](https://discourse.julialang.org/t/sparsearray-types-over-distributed-reduction/53116/2 "2021-01-13T22:20:28Z")

</div>

I looked into this a bit further and it looks like the code responsible for addition in SparseArrays ends up promoting the types.

If the line here [https://github.com/JuliaLang/julia/blob/48795c5e80e92c8cd1790d655eb9cc0012ce8337/stdlib/SparseArrays/src/sparsevector.jl#L1215](https://github.com/JuliaLang/julia/blob/48795c5e80e92c8cd1790d655eb9cc0012ce8337/stdlib/SparseArrays/src/sparsevector.jl#L1215) is switched

from:  
`rind = Vector{Int}(undef, cap)`

to:  
`rind = Vector{promote_type(Tx, Ty)}(undef, cap)`

Then a quick test seems to show that the types stay stable (I did a promote\_type in case of different types mixing).

Is there a reason for this behavior (for example)

```julia
julia> spzeros(Int8, Int8, 3) + spzeros(Int8, Int8, 3)
3-element SparseVector{Int8,Int64} with 0 stored entries

```

or is this perhaps not intended and worth opening an issue for? I don’t want to presume to know the best way to handle the type management.

---

<div class="post-metadata">

**Author:** ![moeddel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/moeddel/32/18641_2.png) [@moeddel](https://discourse.julialang.org/u/moeddel)\
**Post date:** [January 14, 2021, 8:21am UTC](https://discourse.julialang.org/t/sparsearray-types-over-distributed-reduction/53116/3 "2021-01-14T08:21:44Z")

</div>

I am no expert, but I would agree that this at least not necessary, since for simple `SparseVector` addition there is no index arithmetic necessary that could bring an index to overflow. It would mainly be simple index comparison.