# What is the equivalent of this OpenMP Directive in Julia?

**URL:** <https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014>\
**Category:** Performance\
**Tags:** question, multithreading\
**Created:** [May 31, 2022, 8:43pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014 "2022-05-31T20:43:27Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [May 31, 2022, 8:43pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/1 "2022-05-31T20:43:27Z")

</div>

What is the equivalent to the below command in Julia, in which I can run a specific code (or function) on a specific CPU core?

```julia
!$OMP PARALLEL
    # Code
!$OMP END PARALLEL

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [May 31, 2022, 9:31pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/2 "2022-05-31T21:31:57Z")

</div>

> [@Amro](#):
>
> in which I can run a specific code (or function) on a specific CPU core?

That OpenMP directive doesn’t run anything on a “specific CPU core”. It tells OpenMP to execute the given code in parallel using a team of threads (and wait until all threads have completed), but how these are scheduled onto CPU cores is up to the runtime and operating system.

If you just have a block of code in Julia that you want to run N times on all N threads, I guess the closest equivalent would be something like:

```julia
tasks = [Threads.@spawn begin
    # code
end for _ = 1:Threads.nthreads()]
foreach(wait, tasks)

```

For example, running with `julia -t 4` (4 threads):

```julia
julia> tasks = [Threads.@spawn begin
           sleep(10); println("hello from thread ", Threads.threadid())
       end for _ = 1:Threads.nthreads()]
4-element Vector{Task}:
 Task (runnable) @0x000000010a0aa400
 Task (runnable) @0x000000010a0aa570
 Task (runnable) @0x000000010a0aa6e0
 Task (runnable) @0x000000010a0aa850

julia> foreach(wait, tasks)
hello from thread 2
hello from thread 3
hello from thread 4
hello from thread 1

```

---

<div class="post-metadata">

**Author:** ![torrance](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/torrance/32/38990_2.png) [@torrance](https://discourse.julialang.org/u/torrance)\
**Post date:** [June 1, 2022, 5:20am UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/3 "2022-06-01T05:20:43Z")

</div>

Or much more simply:

```
Threads.@threads for i in 1:N
    # Do work
end

```

The `Threads.@threads` macro will distribute the loop into `N / Threads.nthreads()` chunks and assign this work to your active threads. The threads need to be created first, e.g. by starting julia with `--threads [N|auto]`

As @stevengj says, though, this won’t give you the ability to assign to a specific core, although I believe you can set thread affinity by launching julia with the environment variable `JULIA_EXCLUSIVE=1` which might get you some way there (not sure exactly what you’re trying to do there though).

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 1, 2022, 7:02am UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/4 "2022-06-01T07:02:47Z")

</div>

If you want to pin Julia threads to specific cores, you might want to check out [GitHub - carstenbauer/ThreadPinning.jl: Pinning Julia threads to cores](https://github.com/carstenbauer/ThreadPinning.jl) (disclaimer: I’m the author).

If you then want to run code on a specific thread (and core) you can use its `@tspawnat` macro.

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 1, 2022, 7:05pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/5 "2022-06-01T19:05:55Z")

</div>

Thank you very much! Your explanations are really helpful.

> [@stevengj](#):
>
> `sleep(10); println("hello from thread ", Threads.threadid())`

1- I know that `sleep()` is to block the current task for a specified number of seconds. But I didnt know/follow the exact usage in this case (i.e., before `println`), even I have deleted it and got the below. Does it make a barrier? Maybe some comments on the flow of execution of the code would be very helpful.

```julia
julia> tasks = [Threads.@spawn begin
                  println("hello from thread ", Threads.threadid())
                         end for _ = 1:Threads.nthreads()]
hello from thread 2
hello from thread 1
hello from thread 3
hello from thread 4
6-element Vector{Task}:hello from thread 6

 hello from thread 5
Task (done) @0x000000009cef7840
 Task (done) @0x000000009cef7a30
 Task (done) @0x000000009cef7c20
 Task (done) @0x000000009cef7e10
 Task (done) @0x000000009c9cc010
 Task (done) @0x000000009c9cc200

```

2- Since `tasks = [...]` can give the same results, what is the meaning of using `foreach(wait, tasks)`?

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 1, 2022, 7:13pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/6 "2022-06-01T19:13:59Z")

</div>

Thank you very much!

> [@torrance](#):
>
> although I believe you can set thread affinity by launching julia with the environment variable `JULIA_EXCLUSIVE=1` which might get you some way there (not sure exactly what you’re trying to do there though).

Can I put it at the beginning of my julia code as below before running it the REPL?

```julia
ENV["JULIA_EXCLUSIVE"] = 1

```

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 1, 2022, 7:17pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/7 "2022-06-01T19:17:07Z")

</div>

Thanks for your note! Will it be available for Windows soon?

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 1, 2022, 7:38pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/8 "2022-06-01T19:38:53Z")

</div>

> [@Amro](#):
>
> I know that `sleep()` is to block the current task for a specified number of seconds. But I didnt know/follow the exact usage in this case

I only put it there to make each task run for a little while, so that the `foreach(wait, tasks)` had a chance to wait for them to complete. (Otherwise they run so quickly that the tasks are already done by the time I run `wait`.)

> [@Amro](#):
>
> Why the load is not balanced on threads?

Julia uses a dynamic scheduler, not a static schedule. You spawn as many tasks as you want (often many more than you have threads) in order to expose the _potential_ parallelism of your program, and then the runtime scheduler will assign these tasks to threads.

In this particular case, it seems your tasks are running so quickly that the runtime scheduler may have time to migrate them to all the threads.

(One big advantage of a dynamic scheduler is that it can load-balance irregular tasks, whereas a static schedule only works well if you know ahead of time how to equally divide the computational effort. Another big advantage is that parallelism is _composable_ — your code can spawn multiple tasks, and a library routine can also spawn multiple tasks that you don’t know about, and the runtime scheduler will load-balance all of them together.)

> [@Amro](#):
>
> Why `1` is saved in S[1] not 4?

It looks like you never ran anything on thread 1, so `S[1]` is left at whatever you initialized `S` to?

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 1, 2022, 9:14pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/9 "2022-06-01T21:14:58Z")

</div>

> [@stevengj](#):
>
> It looks like you never ran anything on thread 1, so `S[1]` is left at whatever you initialized `S` to?

Oops, I forgot to mention the initialization. For this reason, I putting it again (on 6 threads):

```julia
S = zeros(nthreads());
N = zeros(nthreads());
@threads for i in 1:nthreads()
    println("hello from thread ", threadid())
    S[i] = threadid();
    N[threadid()] = threadid();
end
hello from thread 2
hello from thread 5
hello from thread 6
hello from thread 4
hello from thread 5
hello from thread 3
julia> S
6-element Vector{Float64}:
 3.0
 5.0
 2.0
 6.0
 3.0
 4.0

julia> N
6-element Vector{Float64}:
 0.0
 2.0
 3.0
 4.0
 5.0

```

- From the printed `hello...`, I see that `threadid=5` has addressed two iterations, so number `5` should appear twice in `S` not umber `3`, right?
- Here it is clear that `N[1]=0` because `threadid=1` has not been assigned any iteration.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 1, 2022, 9:32pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/10 "2022-06-01T21:32:22Z")

</div>

> [@Amro](#):
>
> From the printed `hello...`, I see that `threadid=5` has addressed two iterations, so number `5` should appear twice in `S` not umber `3`, right?

I don’t see how you can tell which iteration occurred for which value of `i`, since you didn’t print that (and the `println` statements will _not_ necessarily execute in order of `i`, since they are parallel) … maybe change it to `println("hello $i from thread ", threadid())`.

(Technically, it may be possible for the loop body to migrate from one thread to another while it executes, so you have a bit of a race condition here, but I guess that’s not too likely for such a small/fast loop body.)

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [June 1, 2022, 9:33pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/11 "2022-06-01T21:33:37Z")

</div>

> [@Amro](#):
>
> Will it be available for Windows soon?

Why do you care about the affinity to particular CPUs? My feeling is that this is mostly useful for benchmarking (since it makes the performance more consistent).

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [June 2, 2022, 8:40am UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/12 "2022-06-02T08:40:33Z")

</div>

> [@Amro](#):
>
> Can I put it at the beginning of my julia code as below before running it the REPL?
> 
> ```julia
> ENV["JULIA_EXCLUSIVE"] = 1
> 
> ```

No that won’t work, you need to set the environment variable before starting the Julia process. E.g. `JULIA_EXCLUSIVE=1 julia -t4`

But I agree with Steven here: very likely, you shouldn’t care about the thread-core mapping.

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 2, 2022, 2:13pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/13 "2022-06-02T14:13:46Z")

</div>

> [@stevengj](#):
>
> I don’t see how you can tell which iteration occurred for which value of `i`, since you didn’t print that (and the `println` statements will _not_ necessarily execute in order of `i`, since they are parallel) … maybe change it to `println("hello $i from thread ", threadid())`.

Maybe my understanding for the execution is not correct. Therefore, I made the change based on your comment as below. Please correct me if my explanations for the below results are not true.

- From the `println` statements, thread `2` has executed two iterations when `i` equals `2`&`3`. So, number `2` has to be saved in S[2]&S[3], which is true for S[3] but not for S[2] (=3).
- Following the same logic, S[4] (should =5 not 6), S[5] (should =6 not 5), and S[6] (should =3 not 4).

```julia
S = zeros(nthreads());
@threads for i in 1:nthreads()
    println("hello from $i thread ", threadid())
    S[i] = threadid();
end
hello from 1 thread 4
hello from 3 thread 2
hello from 2 thread 2
hello from 5 thread 6
hello from 4 thread 5
hello from 6 thread 3

julia> S
6-element Vector{Float64}:
 4.0
 3.0
 2.0
 6.0
 5.0
 4.0

```

> [@stevengj](#):
>
> (Technically, it may be possible for the loop body to migrate from one thread to another while it executes, so you have a bit of a race condition here, but I guess that’s not too likely for such a small/fast loop body.)

I think this is what happened above that there is a migration between the `println` and the assignment statements, right?

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 2, 2022, 2:24pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/14 "2022-06-02T14:24:17Z")

</div>

I am thinking to spawn my main code over the `1:nthreads()-1` thread and reserve the last thread to handle the rest code. So, I can reduce the overlap?

---

<div class="post-metadata">

**Author:** ![Amro](https://avatars.discourse-cdn.com/v4/letter/a/4491bb/32.png) [@Amro](https://discourse.julialang.org/u/Amro)\
**Post date:** [June 7, 2022, 7:13pm UTC](https://discourse.julialang.org/t/what-is-the-equivalent-of-this-openmp-directive-in-julia/82014/15 "2022-06-07T19:13:09Z")

</div>

Any feedbacks here please?