### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [June 16, 2019, 6:17am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/2 "2019-06-16T06:17:52Z")
This will heavily depend on your actual application (e.g. Patterns of data transfer and memory access, single machine CPU multi core vs GPU vs cluster).
I don’t think there’s a much better place to start than the docs which you say you’ve read, so if you could maybe post a simple example of an actual bit of Julia code you’re trying to parallelise and any errors you’re getting people might be able to help.
I’m not familiar with C++, but the thing most closely related to a `parallel for` in other languages like MATLAB is `using Distributed; addprocs(2); @distributed for 1:10 print(i) end`
e
---
### Author:  [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)
#### Post date: [June 16, 2019, 7:16am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/3 "2019-06-16T07:16:06Z")
Hi,
the corresponding (multithreaded) recipe in Julia is to use `Threads.@threads` in front of your external loop:
`Threads.@threads for i=0:8`
Under the conditions that the body of the loop is
- parallel (no dependencies)
- long enough to amortize the thread machinery
- not already memory bound
it should give you some speed-up 😉 (if it works with openMP it should work too)
As a usual advice, make sure that you sequential program is already well optimized (type stable, vectorized,…) before entering // business.
---
### Author:  [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)
#### Post date: [June 16, 2019, 7:22am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/4 "2019-06-16T07:22:25Z")
Thank you very much, I added some content to the question.
I have a real problem:
What is the difference between using `Distributed` and the `@threads` mentioned in the [Julia documentation](https://docs.julialang.org/en/v1/manual/parallel-computing/#Multi-Threading-(Experimental)-1)?
---
### Author:  [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)
#### Post date: [June 16, 2019, 7:26am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/5 "2019-06-16T07:26:03Z")
Thank you very much 😃, and i want to know what is the difference between using `Distributed` and `@threads`?
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [June 16, 2019, 7:28am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/6 "2019-06-16T07:28:34Z")
Hello @FujiwaraTakumiEH I have worked in high performance computing for many years.
I am going to say something to you respectfully - think about using the features of Julia to make your code performa faster. then think of threads and parallel.
For instance you can use the @SIMD macro to make use of the vector units on your CPU
And of course there are GPUs.
But first let us ask some Julia experts here to help you speed up the code.
For instance have you looked at broadcasts?
---
### Author:  [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)
#### Post date: [June 16, 2019, 7:29am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/7 "2019-06-16T07:29:49Z")
MT is based on threads while distributed parallelism involves different processes. In the former case all the threads access a **shared** memory. MT is well adapted to SMP computers (single nodes) and can be faster (and trickier) and consume less memory than distributed parallelism. It also can handle finer grained parallelism. On the other hand, distributed parallelism can run on several nodes (cluster) and can tackle larger scales.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [June 16, 2019, 7:31am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/8 "2019-06-16T07:31:20Z")
I will also give my explanation of threads and Distributed.
Threads are lightweight processes - they share the same memory space as other threads.
So you have to be careful about one thread corrupting the data which another thread is working with.
Threads only work within one compute server - they are schedule by the operating system.
Distributed processes are more coarse grained. But they can scale across compute servers.
They way I understand it is Julia distributed processes are separate Julia instances, started by an ssh process.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [June 16, 2019, 7:36am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/9 "2019-06-16T07:36:41Z")
Rhis is an article concerning vectorisation in Julia.
I think your loop should vectorise well - each i is independent.
[https://juliacomputing.com/blog/2017/09/27/auto-vectorization-in-julia.html](https://juliacomputing.com/blog/2017/09/27/auto-vectorization-in-julia.html)
You also consider the type of the array which you use. Is N a fixed size?
> [@Whats the difference between a regular array and an array from StaticArrays](https://discourse.julialang.org/t/whats-the-difference-between-a-regular-array-and-an-array-from-staticarrays/14454/2):
>
> The type of a regular Julia Array only includes the number of dimensions, not the size: julia\> typeof(Array{Int64}(3)) Array{Int64,1} julia\> typeof(Array{Int64}(10)) Array{Int64,1} julia\> typeof(Array{Int64}(1000000)) Array{Int64,1} so the difference between a StaticArray and an Array is exactly what the docs here say: [https://github.com/JuliaArrays/StaticArrays.jl](https://github.com/JuliaArrays/StaticArrays.jl) , namely that the size of a StaticArray is actually part of the type: julia\> typeof(SVector(0, 0)) StaticArrays.SArray{Tuple{2}…
---
### Author:  [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)
#### Post date: [June 16, 2019, 7:46am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/10 "2019-06-16T07:46:46Z")
In addition to John’s advices you may try (still experimental) @ffevote tool:
[GFlops.jl](https://github.com/triscale-innov/GFlops.jl)
before using any parallelism (it does not work with MT). It allows to compute the actual computational speed of your implementation (in GFlops) so that you can estimate its efficiency.
---
### Author:  [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)
#### Post date: [June 16, 2019, 7:52am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/11 "2019-06-16T07:52:15Z")
Thank you very much for your reply. 😁 I have used some of the `broadcasts` in programming, as well as the `map` function. Finally, the main part of this program can only be written in the format of a `for` loop. I have two specific questions:
1. Is this function using boardcast or map mean that it is already parallel?
2. for example, in my use case, I can only use multiple cores on one computer for parallel computing. Then, if the settings are correct, `@threads` should be more efficient than `Distributed`?
---
### Author:  [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)
#### Post date: [June 16, 2019, 7:58am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/12 "2019-06-16T07:58:58Z")
1. No but yes. broadcast or map are not parallel but may produce a vectorized binary (SIMD) that is a special form of parallelism.
2. Yes, in principle if you don’t need to scale between several nodes/machines the `@thread` overhead is much lower than distributed and should be faster.
---
### Author:  [@pszufe](https://discourse.julialang.org/u/pszufe)
#### Post date: [June 16, 2019, 10:25am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/13 "2019-06-16T10:25:34Z")
Threading support in Julia is experimental and in practice it means that writing a production parallel code is difficult due to various glitches. The easiest piece of code that crashes a multi-threaded Julia code is following: `Threads.@threads for i in 1:10 sleep(1) end`. There are also hard to manage compiler chase issues (two threads starting to compile the same function at the same time) that also occasionally result in a crash.
However support for multi-processing and distributed computing in Julia is brilliant. Depending on your scenario look at the following info when learning:
- Start with docs [Parallel Computing · The Julia Language](https://docs.julialang.org/en/v1/manual/parallel-computing/index.html) . You have to learn carefully green threading macros because you need them to control your distributed processes, you can skip multi-threading for the reasons above.
- two main blocks for any distributed Julia code are `@distributed` and `pmap` - learn them carefully
- have a look at ParallelDataTransfer.jl - and learn it along `@spawnat` macro and `fetch` function. This macro is usually used along with the green threading mentioned above.
- now (depending on your needs) have a look at the following packages: SharedArrays.jl (many processes sharing data on a local machine), DistributedArrays.jl (array distributed over many local or remote processes)
- for some data analytics jobs you might also consider tools such as JuliaDB.
Hope this guide helps!
---
### Author:  [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)
#### Post date: [June 16, 2019, 10:37am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/14 "2019-06-16T10:37:41Z")
BTW, what is the PARTR status ? A composable a safe nested //ism would make Julia even more (is that possible ?) attractive 😉
---
### Author:  [@pszufe](https://discourse.julialang.org/u/pszufe)
#### Post date: [June 16, 2019, 12:13pm UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/15 "2019-06-16T12:13:39Z")
Regarding PARTR This is the post to track 🙂
> [@Compiler work priorities](https://discourse.julialang.org/t/compiler-work-priorities/17623):
>
> In response to I thought I might write a little post about the rough priorities of the compiler team: Correctness finding and fixing compiler and inference bugs Multithreading [non-copying task stack switching](https://github.com/JuliaLang/julia/pull/13099) (done) [new PARTR parallel runtime](https://github.com/JuliaLang/julia/pull/22631)[locks for I/O operations](https://github.com/libuv/libuv/issues/1595) other thread safety Compile-time latency, aka “the time-to-first-plot problem” making compilation faster caching more things Compiler-related packages and tools [PackageCompiler](https://github.com/JuliaLang/PackageCompiler.jl) Debugger [Cxx](https://github.com/Keno/Cxx.jl) type checking/linting J…
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [June 16, 2019, 2:08pm UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/16 "2019-06-16T14:08:05Z")
Implemented on master, lacking only an official API.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [June 16, 2019, 8:09pm UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/17 "2019-06-16T20:09:02Z")
I am really interested in Gflops.jl
With Intel CPUs you have counters which will return the exact number of floating point operation instructions issued and retired. These are different due to speculative exectution. The brings in discussions of Spectre, Meltdown exploits.
However those counters are specific to Intel - we live in a worls with AMD, ARM and other CPUs of course.
---
### Author:  [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)
#### Post date: [June 17, 2019, 7:51am UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/18 "2019-06-17T07:51:36Z")
FWIW, you could take a look at [https://github.com/crstnbr/julia-workshop/blob/master/5%20Parallel%20computing/parallel-computing.ipynb](https://github.com/crstnbr/julia-workshop/blob/master/5%20Parallel%20computing/parallel-computing.ipynb) (an overview/tutorial I once prepared).
---
### Author:  [@FujiwaraTakumiEH](https://discourse.julialang.org/u/FujiwaraTakumiEH)
#### Post date: [June 22, 2019, 2:01pm UTC](https://discourse.julialang.org/t/questions-about-getting-started-with-parallel-computing/25341/19 "2019-06-22T14:01:03Z")
Hello 😄, after reading your link, I would like to ask the following questions:
In your `ipynb` file you have the following:
```julia
Distributed loop, but no reduction
The following example might not be doing what you'd expect it to. Why?
```
```julia
a = zeros(10)
@distributed for i = 1:10
a[i] = i
end
```
Note that `@distributed` without a reduction function returns a `Task` . It is basically a distributed version of `@spawn` for all the iterations.
* * *
I don’t know how to solve this problem.😅
1. Do you mean using `SharedArrays` to solve this problem?
2. If I want to add an element to an array that doesn’t know the size, what should I do? I tried the following code but it didn’t work.
> No parallel situation:
```julia
function f(n)
ax = Vector{Float64}()
for i = 1:n
append!(ax,i)
end
return ax
end
f(5)
5-element Array{Float64,1}:
1.0
2.0
3.0
4.0
5.0
```
> Parallel situation:
```julia
function f(n)
ax = SharedArray{Float64}()
@distributed for i = 1:n
append!(ax,i)
end
return ax
end
f(5)
0-dimensional SharedArray{Float64,0}:
0.0
```
I think `ShareArrays` should pre-define the memory area of the specified size in memory, so there may be no `append!` in `ShareArrays`, but if I want to achieve my purpose (add elements to an array of unknown size and use parallel Method) What should I do?