# ANN: Data parallelism tutorial

**URL:** <https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824>\
**Category:** General Usage\
**Tags:** announcement, parallel, multithreading, tutorials\
**Created:** [October 6, 2020, 2:04am UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824 "2020-10-06T02:04:50Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [October 6, 2020, 2:04am UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/1 "2020-10-06T02:04:50Z")

</div>

Hi, I just wrote [A quick introduction to data parallelism in Julia](https://juliafolds.github.io/data-parallelism/tutorials/quick-introduction/)!

For a quick flavor of the tutorial, here is the table of contents:

> 1. Getting julia and libraries
> 2. Starting julia
> 1. Starting julia with multiple worker processes
> 
> 3. Mapping
> 1. Practical example: Stopping time of Collatz function
> 
> 4. Iterator comprehensions
> 5. Pre-defined reductions
> 1. Practical example: Maximum stopping time of Collatz function
> 2. OnlineStats.jl
> 
> 6. Manual reductions
> 1. Parallel findmin/findmax with `@reduce() do`
> 2. Parallel findmin/findmax with `ThreadsX.reduce` (tedious!)
> 3. Histogram with reduce
> 4. Practical example: Histogram of stopping time of Collatz function
> 5. Quick notes on `@threads` and `@distributed`
> 
> 7. Next steps

Although Julia since the release of 1.3 has been a wonderful playground for parallel computing, there is no easy-to-access entry-level resource for data-parallel programming. As a result, Q&A in discourse often focuses on non-composable `@threads` or low-level `@spawn` and sometimes with some sub-optimal/questionable coding patterns (please [no locks/atomics](https://juliafolds.github.io/data-parallelism/tutorials/quick-introduction/#no-locks) for sum!). I’ve been building up tooling for data parallelism in [JuliaFolds](https://github.com/JuliaFolds) but now it’s a bit scattered across a few packages and hard to get a big picture of it. So, I’m hoping that a quick tutorial is helpful for this.

If you have a question, feedback, request for new topics, or any comment, please feel free to post it here or [suggest a change on GitHub](https://github.com/JuliaFolds/data-parallelism/blob/master/src/tutorials/quick-introduction.md) or [open an issue](https://github.com/JuliaFolds/data-parallelism/issues)!

---

<div class="post-metadata">

**Author:** ![johnh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johnh/32/3615_2.png) [@johnh](https://discourse.julialang.org/u/johnh)\
**Post date:** [October 6, 2020, 7:05am UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/2 "2020-10-06T07:05:47Z")

</div>

A READMEfile on the top level Juliafolds page would be useful.  
Not being rude but if I land on a page like that and here is not a clear description of what I can find in the repositories I will rapidly leave. Sorry - really not being rude.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [October 6, 2020, 7:29am UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/3 "2020-10-06T07:29:33Z")

</div>

That’s a good suggestion! Thanks! I don’t know how to use README for organization profile page so I just tweaked the pinned packages and added a quick description.

(From a quick googling, I can find [how to use README for usesr profile](https://docs.github.com/en/free-pro-team@latest/github/setting-up-and-managing-your-github-profile/managing-your-profile-readme) but it looks like [it’s not usable for organization](https://github.community/t/readme-for-organization-front-page/2920) as of Feb 2020.)

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [October 6, 2020, 7:41am UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/4 "2020-10-06T07:41:51Z")

</div>

Thank you !

I also like this table [Libraries for parallelism in Julia](https://juliafolds.github.io/data-parallelism/explanation/libraries/#libraries_for_parallelism_in_julia)✅

> request

Would it be possible to add more visualisation ?

imho:

- it can be helpful to understand new paradigm ideas or concepts

My favorite example:  
[Golang Visualization examples (divan.dev)](https://divan.dev/posts/go_concurrency_visualize/)

 ![WebGL animation](https://global.discourse-cdn.com/julialang/original/3X/0/2/022b4128c069d026e27be0e8c298130761bb9def.gif)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [October 6, 2020, 8:19am UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/5 "2020-10-06T08:19:05Z")

</div>

> [@ImreSamu](#):
>
> I also like this table [Libraries for parallelism in Julia](https://juliafolds.github.io/data-parallelism/explanation/libraries/#libraries_for_parallelism_in_julia) ✅

Haha, thanks. That’s me playing with Franklin.jl as it’s the first time I use it. It’s a fantastic tool for something like this.

> [@ImreSamu](#):
>
> My favorite example:  
> [Golang Visualization examples (divan.dev)](https://divan.dev/posts/go_concurrency_visualize/)

Thanks for the link. Wow, there are tons of cool visuals. I’ll watch the GopherCon talk!

> [@ImreSamu](#):
>
> Would it be possible to add more visualisation ?

By visualization, do you mean some kind of task or call tree? For example, [parallel `collect` tutorial in Transducers.jl](https://juliafolds.github.io/Transducers.jl/dev/tutorials/tutorial_parallel/#tutorial-parallel-collect) has this low-tech diagram:

> ```julia
> [1,2,3,4] <------------- append!!([1,2], [3,4]) == [1,2,3,4]
> / \
> [1,2] [3,4] <------- append!!([3], [4]) == [3, 4]
> / \ / \
> [1] [2] [3] [4] <---- append!!([], [4]) == [4]
> / \ / \ / \ / \
> [] [1] [] [2] [] [3] [] [4]
> 
> ```

Are you thinking something like this?

I think it’s definitely a good idea to have a low-level explanation like this in a different more in-depth tutorial. But I am a bit torn for mentioning something like this the introduction-level tutorial. Ideally, a casual user does not have to care about a low-level execution strategy. But I don’t think the current version is perfect in this respect since I talk about implementation a bit for explaining how `@floop` works.

Also, I think it’s important to note that [concurrency is not parallelism](https://blog.golang.org/waza-talk). In a way, concurrency is all about controlling how things are executed. So, it is important to know about how tasks interact and having a mental model via visualization is very useful. On the other hand, (high-level) data parallelism is all about _not_ thinking about how things are executed and let the libraries (and possibly the compilers) find an adequate execution strategy.

Obviously, there are multiple ways to start understanding data parallelism, and starting from a very low-level description could be a good option for some people. Probably the best way is to provide a few different tutorials for people with different tastes.

---

<div class="post-metadata">

**Author:** ![PetrKryslUCSD](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/petrkryslucsd/32/215825_2.png) [@PetrKryslUCSD](https://discourse.julialang.org/u/PetrKryslUCSD)\
**Post date:** [October 6, 2020, 4:56pm UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/6 "2020-10-06T16:56:57Z")

</div>

This was precisely my point in [https://discourse.julialang.org/t/the-juliadebug-repo-how-to-be-helpful-to-newbies-and-old-hands-as-well/47618:](https://discourse.julialang.org/t/the-juliadebug-repo-how-to-be-helpful-to-newbies-and-old-hands-as-well/47618:) I really don’t see a way of posting a general information splash at the top of the organization landing page. That is a very unfortunate omission which makes organizations on github much less useful than they could be.

This was the reason I conceived of the JuliaPDE organization as consisting of a single repo, the survey of the PDE-development landscape. This makes the only repo act as a kind of guide (readme).

---

<div class="post-metadata">

**Author:** ![ImreSamu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/imresamu/32/20677_2.png) [@ImreSamu](https://discourse.julialang.org/u/ImreSamu)\
**Post date:** [October 6, 2020, 8:49pm UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/7 "2020-10-06T20:49:09Z")

</div>

> By visualization, do you mean some kind of task or call tree?

nothing concrete …  
Something : _ **Eye Candy** _ + Useful  
So the first impression should be : it is interesting + Cool + I want to learn …

So my original comment … just a trigger for brainstorming …

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [October 6, 2020, 9:00pm UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/8 "2020-10-06T21:00:32Z")

</div>

> [@ImreSamu](#):
>
> So the first impression should be : it is interesting + Cool + I want to learn …

Yeah, I totally get it! That’s kinda why I put the plots about Collatz conjecture even though it’s nothing to do with parallelism 🙂

(OK, to be honest, the initial plan was to discuss how to use `basesize` for load-balancing parallel reduction. Since I knew the stopping time ≈ compute time is unpredictable and variable, I thought it might be interesting to show the effect of `basesize` on the run-time performance. It turned out the stopping time does not have “fat enough” tail to let me discuss meaningful load-balancing.)

---

<div class="post-metadata">

**Author:** ![etorkia](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/etorkia/32/11677_2.png) [@etorkia](https://discourse.julialang.org/u/etorkia)\
**Post date:** [October 8, 2020, 6:33pm UTC](https://discourse.julialang.org/t/ann-data-parallelism-tutorial/47824/9 "2020-10-08T18:33:44Z")

</div>

thanks for sharing…the tutorial looks awesome.