ANN: Data parallelism tutorial

General Usage
, , ,
1

Hi, I just wrote A quick introduction to data parallelism in Julia!

For a quick flavor of the tutorial, here is the table of contents:

  1. Getting julia and libraries
  2. Starting julia
    1. Starting julia with multiple worker processes
  3. Mapping
    1. Practical example: Stopping time of Collatz function
  4. Iterator comprehensions
  5. Pre-defined reductions
    1. Practical example: Maximum stopping time of Collatz function
    2. OnlineStats.jl
  6. Manual reductions
    1. Parallel findmin/findmax with @reduce() do
    2. Parallel findmin/findmax with ThreadsX.reduce (tedious!)
    3. Histogram with reduce
    4. Practical example: Histogram of stopping time of Collatz function
    5. Quick notes on @threads and @distributed
  7. Next steps

Although Julia since the release of 1.3 has been a wonderful playground for parallel computing, there is no easy-to-access entry-level resource for data-parallel programming. As a result, Q&A in discourse often focuses on non-composable @threads or low-level @spawn and sometimes with some sub-optimal/questionable coding patterns (please no locks/atomics for sum!). I’ve been building up tooling for data parallelism in JuliaFolds but now it’s a bit scattered across a few packages and hard to get a big picture of it. So, I’m hoping that a quick tutorial is helpful for this.

If you have a question, feedback, request for new topics, or any comment, please feel free to post it here or suggest a change on GitHub or open an issue!

45 Likes
2

A READMEfile on the top level Juliafolds page would be useful.
Not being rude but if I land on a page like that and here is not a clear description of what I can find in the repositories I will rapidly leave. Sorry - really not being rude.

2 Likes
3

That’s a good suggestion! Thanks! I don’t know how to use README for organization profile page so I just tweaked the pinned packages and added a quick description.

(From a quick googling, I can find how to use README for usesr profile but it looks like it’s not usable for organization as of Feb 2020.)

4

Thank you !

I also like this table Libraries for parallelism in Julia :white_check_mark:

request

Would it be possible to add more visualisation ?

imho:

  • it can be helpful to understand new paradigm ideas or concepts

My favorite example:
Golang Visualization examples (divan.dev)

WebGL animation
WebGL animation760×434 1.29 MB

2 Likes
5

Haha, thanks. That’s me playing with Franklin.jl as it’s the first time I use it. It’s a fantastic tool for something like this.

Thanks for the link. Wow, there are tons of cool visuals. I’ll watch the GopherCon talk!

By visualization, do you mean some kind of task or call tree? For example, parallel collect tutorial in Transducers.jl has this low-tech diagram:

          [1,2,3,4] <------------- append!!([1,2], [3,4]) == [1,2,3,4]
         /         \
    [1,2]           [3,4] <------- append!!([3], [4]) == [3, 4]
   /     \         /     \
 [1]     [2]     [3]     [4] <---- append!!([], [4]) == [4]
 / \     / \     / \     / \
[] [1]  [] [2]  [] [3]  [] [4]

Are you thinking something like this?

I think it’s definitely a good idea to have a low-level explanation like this in a different more in-depth tutorial. But I am a bit torn for mentioning something like this the introduction-level tutorial. Ideally, a casual user does not have to care about a low-level execution strategy. But I don’t think the current version is perfect in this respect since I talk about implementation a bit for explaining how @floop works.

Also, I think it’s important to note that concurrency is not parallelism. In a way, concurrency is all about controlling how things are executed. So, it is important to know about how tasks interact and having a mental model via visualization is very useful. On the other hand, (high-level) data parallelism is all about not thinking about how things are executed and let the libraries (and possibly the compilers) find an adequate execution strategy.

Obviously, there are multiple ways to start understanding data parallelism, and starting from a very low-level description could be a good option for some people. Probably the best way is to provide a few different tutorials for people with different tastes.

7 Likes
6

This was precisely my point in https://discourse.julialang.org/t/the-juliadebug-repo-how-to-be-helpful-to-newbies-and-old-hands-as-well/47618: I really don’t see a way of posting a general information splash at the top of the organization landing page. That is a very unfortunate omission which makes organizations on github much less useful than they could be.

This was the reason I conceived of the JuliaPDE organization as consisting of a single repo, the survey of the PDE-development landscape. This makes the only repo act as a kind of guide (readme).

7

By visualization, do you mean some kind of task or call tree?

nothing concrete …
Something : Eye Candy + Useful
So the first impression should be : it is interesting + Cool + I want to learn …

So my original comment … just a trigger for brainstorming …

8

Yeah, I totally get it! That’s kinda why I put the plots about Collatz conjecture even though it’s nothing to do with parallelism :slight_smile:

(OK, to be honest, the initial plan was to discuss how to use basesize for load-balancing parallel reduction. Since I knew the stopping time ≈ compute time is unpredictable and variable, I thought it might be interesting to show the effect of basesize on the run-time performance. It turned out the stopping time does not have “fat enough” tail to let me discuss meaningful load-balancing.)

1 Like
9

thanks for sharing…the tutorial looks awesome.

1 Like