# Mixing vectorization and parallelization

**URL:** <https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800>\
**Category:** General Usage\
**Created:** [March 18, 2018, 10:06pm UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800 "2018-03-18T22:06:30Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [March 18, 2018, 10:06pm UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800/1 "2018-03-18T22:06:30Z")

</div>

I have a Monte Carlo computation that involves computing an ensemble average of random vectors. If I were to compute this using vectorization, it would look like:

```julia
Ybar = zeros(d);
for j = 1:nsamples
    @. Ybar += f(randn())/nsamples;
end

```

where `f` is some vector valued function. In contrast, using shared memory parallization, I would do

```julia
Ybar = @parallel(+) for j=1:nsamples
    f(randn())/nsamples;
end

```

Do people have any recommendations for how to maximize the efficiency of these kinds of computations?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [March 20, 2018, 12:29pm UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800/2 "2018-03-20T12:29:22Z")

</div>

If the operation is such that you can calculate statistics for parts of the data then merge (eg a mean), then you can do that. This may help:

> **[GitHub - joshday/OnlineStats.jl: ⚡ Single-pass algorithms for statistics](https://github.com/joshday/OnlineStats.jl)**
>
> ⚡ Single-pass algorithms for statistics. Contribute to joshday/OnlineStats.jl development by creating an account on GitHub.

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [March 20, 2018, 1:53pm UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800/3 "2018-03-20T13:53:12Z")

</div>

> [@gideonsimpson](#):
>
> Ybar = @parallel(+) for j=1:nsamples  
> f(randn())/nsamples;  
> end

That’s not shared memory parallelism. You would want to use `Threads.@threads`, storing an intermediate sum for each thread then summing the intermediates.

Also the above 2 code snippets give different results. The first returns a vector and the second returns a number, so I don’t see how the comparison makes sense. See [How to add arrays using multithreading?](https://discourse.julialang.org/t/how-to-add-arrays-using-multithreading/9532) for doing the second example’s job with threading.

---

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [March 20, 2018, 5:03pm UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800/4 "2018-03-20T17:03:30Z")

</div>

So I see that this is not actually shared memory parallelism. I was really just referring to working on a shared memory machine.  
Regarding your second point, if f(x):\mathbb{R}\to \mathbb{R}^d, is there no way to get `@parallel(+)` to do the desired reduction?

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [March 21, 2018, 12:23am UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800/5 "2018-03-21T00:23:13Z")

</div>

Oh I misunderstood. If `f` returns a vector, then it should just work as expected, and then both examples would be doing the same. Broadcasting helps with doing inplace operations to avoid allocations, but to do reduction, data has to be sent between processors anyways so when it is received I believe it gets allocated in the respective processor’s memory. So are you asking if you can avoid allocating an output vector while summing the individual results after they get received? Try defining an inplace function that wraps `+` and use that in the reduction instead. It can overwrite the first argument and then return it, so as to avoid allocation of the output. The arguments to the reduction are not used after reduction so I think it should be fine.

---

<div class="post-metadata">

**Author:** ![gideonsimpson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gideonsimpson/32/1928_2.png) [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Post date:** [March 23, 2018, 4:18pm UTC](https://discourse.julialang.org/t/mixing-vectorization-and-parallelization/9800/6 "2018-03-23T16:18:22Z")

</div>

Yea, I was basically asking what was the “clever” way to do a `parallel(+)` reduction when the reduced variable was vector/array valued.