# Have a time issue, need advises on my parallel code

**URL:** <https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686>\
**Category:** Performance\
**Tags:** parallel\
**Created:** [March 13, 2018, 3:07pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686 "2018-03-13T15:07:30Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ersoacc](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@ersoacc](https://discourse.julialang.org/u/ersoacc)\
**Post date:** [March 13, 2018, 3:07pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/1 "2018-03-13T15:07:30Z")

</div>

Hello everyone.

I am working on a computation fluid dynamics code which needs to solve some subdomains parallel defined in function _ASPINS1Sub_.  
To do this, I wrote the code given below which consists of many _if statements_.

```julia
g1 = @spawnat 2 ASPINS1Sub(COMP, u, Fparam, N, Tln, UTln, MIln, SOLll,
            Tll, UTll, Sis, Sin, 1, G, INs1)
if nsubs > 1
                g2 = @spawnat 3 ASPINS1Sub(COMP, u, Fparam, N, Tln, UTln, MIln, SOLll,
                Tll, UTll, Sis, Sin, 2, G, INs1) end
 if nsubs > 2
                g3 = @spawnat 4 ASPINS1Sub(COMP, u, Fparam, N, Tln, UTln, MIln, SOLll,
                Tll, UTll, Sis, Sin, 3, G, INs1) end
if nsubs > 3
                g4 = @spawnat 5 ASPINS1Sub(COMP, u, Fparam, N, Tln, UTln, MIln, SOLll,
                Tll, UTll, Sis, Sin, 4, G, INs1) end
.
.
.

```

nsubs is number of subdomains. So I write many if2 like this.  
At the end I collect data and sum them as:

```julia
G = fetch(g1)
if nsubs > 1 G += fetch(g2) end
if nsubs > 2 G += fetch(g3) end
if nsubs > 3 G += fetch(g4) end
if nsubs > 4 G += fetch(g5) end
.
.
.

```

I also have 2 functions like in this patterns, many if statements.  
However I have an issue with having too much if statemets.  
It makes it take many time to enter to 2 upper function, 30-60 sec.

There should be an easier/alternative way to do this.  
I need advises to make it better.

---

<div class="post-metadata">

**Author:** ![stillyslalom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stillyslalom/32/45687_2.png) [@stillyslalom](https://discourse.julialang.org/u/stillyslalom)\
**Post date:** [March 13, 2018, 10:21pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/2 "2018-03-13T22:21:27Z")

</div>

It’s hard to determine how to structure the parallelism without knowing what your solver’s doing. I’m having a hard time imagining a fluids code where you’d simply sum the results from each subdomain. Can you put together a [minimal working example](https://stackoverflow.com/help/mcve)?

This section of the manual may be helpful: [https://docs.julialang.org/en/stable/manual/parallel-computing#man-shared-arrays-1](https://docs.julialang.org/en/stable/manual/parallel-computing#man-shared-arrays-1)

---

<div class="post-metadata">

**Author:** ![greg\_plowman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/greg_plowman/32/8100_2.png) [@greg\_plowman](https://discourse.julialang.org/u/greg_plowman)\
**Post date:** [March 14, 2018, 1:18am UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/3 "2018-03-14T01:18:13Z")

</div>

If you really want a parallel reduction, you can use `@parallel` or `pmap`.

See [https://docs.julialang.org/en/stable/manual/parallel-computing#Parallel-Map-and-Loops-1](https://docs.julialang.org/en/stable/manual/parallel-computing#Parallel-Map-and-Loops-1)

Something like:

```julia
function f(nsubs)
    Gsum = @parallel (+) for subdomain in 1:nsubs
        ASPINS1Sub(COMP, u, Fparam, N, Tln, UTln, MIln, SOLll, Tll, UTll, Sis, Sin, subdomain, G, INs1)
    end
    return Gsum
end

function g(nsubs)
    Gs = pmap(subdomain -> ASPINS1Sub(COMP, u, Fparam, N, Tln, UTln, MIln, SOLll, Tll, UTll, Sis, Sin, subdomain, G, INs1), 1:nsubs)
    return sum(Gs)
end

```

---

<div class="post-metadata">

**Author:** ![ersoacc](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@ersoacc](https://discourse.julialang.org/u/ersoacc)\
**Post date:** [March 14, 2018, 12:33pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/4 "2018-03-14T12:33:13Z")

</div>

Thank you. I have checked that page and this worked well. I finally got rid of many if statements.

---

<div class="post-metadata">

**Author:** ![ersoacc](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@ersoacc](https://discourse.julialang.org/u/ersoacc)\
**Post date:** [March 14, 2018, 12:33pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/5 "2018-03-14T12:33:18Z")

</div>

The code is a kind of nonlinear preconditioner. I have fixed that issue thanks to greg\_plowman.

On the other hand, I also need advise on Shared Arrays.  
In one routine, I am generationg very large and sparse matrix in a for loop. I tried to parallelise but couldnt do it because of that fact that ın Julia there is no Shared Sparse Matrix.  
I have seen that a package([ParallelSparseMatMul](https://github.com/madeleineudell/ParallelSparseMatMul.jl) support it but package did not worked for me.

Any ideas on this?

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [March 14, 2018, 1:58pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/6 "2018-03-14T13:58:28Z")

</div>

Is your plan to assemble the sparse matrix in parallel? I advise against it (sorry if misread your question).

Remember that changing the sparsity structure of sparse matrices is O(N); hence, get an O(N^2) algorithm if you generate your matrix by something like

```julia
A=sparse([],[],Float64[],100,100);
for idx,val in keyvals
A[idx...]=val
end

```

You should rather collect the keys and values and call a sparse matrix constructor `sparse(Is, Js, Vs, m, n)` on it. I think there is no value in trying to assemble in parallel; the built-in single-threaded algorithm should be able to fill all your available memory very rapidly. If the computation of the coefficients is the expensive part, then just store them somewhere and join the sets before calling the constructors; if the computation of sparsity structure is cheap, you can also use the next paragraph.

Of course, mutating the sparse matrix post-construction in multiple threads makes perfect sense, as long as you never modify the sparsity structure. In this case, make sure to create slots for entries which are initially zero but might become nonzero later (by having an `i,j,0.0` entry during construction). I am not sure about modifying values in different processes (`@parallel` instead of `@threads`).

If you are instead asking for parallel sparse BLAS, sorry, no idea.

---

<div class="post-metadata">

**Author:** ![vvjn](https://avatars.discourse-cdn.com/v4/letter/v/5f9b8f/32.png) [@vvjn](https://discourse.julialang.org/u/vvjn)\
**Post date:** [March 14, 2018, 2:03pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/7 "2018-03-14T14:03:25Z")

</div>

Dagger ([GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel execution](https://github.com/JuliaParallel/Dagger.jl)) has support for distributed sparse matrices. Also JuliaDB.jl has a distributed n-dimensional sparse array, but I don’t think it has the typical array operations implemented on it since it’s used in the context of databases.

---

<div class="post-metadata">

**Author:** ![ersoacc](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@ersoacc](https://discourse.julialang.org/u/ersoacc)\
**Post date:** [March 18, 2018, 2:50pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/8 "2018-03-18T14:50:40Z")

</div>

I have checked Dagger package. I saw the only usage is s1 = sprand(Blocks(4000,4000), 10^4, 10^4, 0.01).  
I could not impelement it on my usage that is:

A = spzeros(N,N)  
then, fill some of the element of A within a for loop.

The pattern of A matrix is not known and is changable for grid size.

---

<div class="post-metadata">

**Author:** ![vvjn](https://avatars.discourse-cdn.com/v4/letter/v/5f9b8f/32.png) [@vvjn](https://discourse.julialang.org/u/vvjn)\
**Post date:** [March 18, 2018, 7:19pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/9 "2018-03-18T19:19:44Z")

</div>

Typically when constructing sparse matrices, you would accumulate the indices and values in separate arrays and then construct the matrix at the end like foobar said above. But if you want to construct a sparse array using getindex and setindex!, the SparseMatrixCSC data structure will be slow since setindex! is an O(nnz) operation for it. I made a sparse matrix datastructure ([https://github.com/vvjn/SparseMatrixLILs.jl](https://github.com/vvjn/SparseMatrixLILs.jl)) a while ago that does setindex! in O(d) where d is the average number of elements in a column. It’s good for constructing sparse matrices conveniently. I guess if Dagger had setindex! it could do what you’re thinking of.

---

<div class="post-metadata">

**Author:** ![Juser](https://avatars.discourse-cdn.com/v4/letter/j/34f0e0/32.png) [@Juser](https://discourse.julialang.org/u/Juser)\
**Post date:** [March 19, 2018, 9:01pm UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/10 "2018-03-19T21:01:03Z")

</div>

As other suggested, you should assemble the [i,j,v] rows (i and j are indices and v are values) in parallel (one matrix on each processor) and then vertically concatenate them on the master process and convert that matrix into a sparse matrix. Thus, most of the sparse matrix creation can be done in parallel while only the last conversion is done by a single processor. (This last part could be made more efficient if there is a particular known sparsity structure such as the matrix being block-diagonal.)

---

<div class="post-metadata">

**Author:** ![ersoacc](https://avatars.discourse-cdn.com/v4/letter/e/9d8465/32.png) [@ersoacc](https://discourse.julialang.org/u/ersoacc)\
**Post date:** [March 20, 2018, 7:50am UTC](https://discourse.julialang.org/t/have-a-time-issue-need-advises-on-my-parallel-code/9686/11 "2018-03-20T07:50:35Z")

</div>

Thank you all.

I better try this. I understand what you mean.