# With ParallelStencil, is it possible to launch multiple kernels and sync later?

**URL:** <https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843>\
**Category:** GPU\
**Created:** [April 8, 2025, 1:04pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843 "2025-04-08T13:04:39Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![pchaubal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pchaubal/32/213778_2.png) [@pchaubal](https://discourse.julialang.org/u/pchaubal)\
**Post date:** [April 8, 2025, 1:04pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/1 "2025-04-08T13:04:39Z")

</div>

I have written a code to run on GPUs with ParallelStencil but I don’t see a significant speedup by moving to GPUs. I am not sure whether I am using the framework as intended and there exists an easy fix for this. So, any help to get more performance would be greatly appreciated.

To illustrate my problem, I have a minimal example code below. The code launches 2 big kernels (UpdateA ! and UpdateB!) which are supposed to do most of the computation, and a few of small kernels which do only a fraction of actual computation. The code ends up spending lot more time on the smaller kernels.

```julia

const USE_GPU = true
# using BenchmarkTools
using ParallelStencil
using ParallelStencil.FiniteDifferences2D

@static if USE_GPU
    @init_parallel_stencil(CUDA, Float64, 2);
else
    @init_parallel_stencil(Threads, Float64, 2);
end

function main2D()
    # Numerics
    nx, ny = 1024, 512; # Number of gridpoints in dimensions x and y
    nt = 10000; # Number of time steps
    c0 = 10.0

    # Array initializations
    A = @zeros(nx, ny);
    B = @zeros(nx, ny);
    # A2 = @zeros(nx, ny);
    C = @rand(nx, ny);

    # Initial conditions
    A .= 1.5;
 
    # Time loop
    dt = 1/nt;
    for it = 1:nt
        if (it == 11)
            GC.enable(false)
            global t_tic=time() # Start measuring time.
        end

        @parallel UpdateA!(A, B, C)

        # the for loops are just to launch many small kernels
        # In actual code, these are different small kernels not called in loop
        for i in 1:10:100
            @parallel (i:i+10, 1:ny) ASubset!(A)
        end

        @parallel UpdateB!(A, B, C)

        # the for loops are just to launch many small kernels
        # In actual code, these are different small kernels not called in loop
        for i in 1:10:300
            @parallel (1:nx, i:i+10) BSubset!(B)
        end
    end
    time_s = time() - t_tic

end

@parallel_indices (ix, iy) function UpdateA!(A, B, C)
    A[ix, iy] = A[ix, iy] + C[ix, iy] * B[ix, iy]
    return
end

@parallel_indices (ix, iy) function UpdateB!(A, B, C)
    B[ix, iy] = B[ix, iy] + C[ix, iy] * A[ix, iy]
    return
end

@parallel_indices (ix, iy) function ASubset!(A)
    # Reduce the value by 10%
    A[ix, iy] = 0.9*A[ix, iy]
    return
end

@parallel_indices (ix, iy) function BSubset!(B)
    # Increase the value by 10%
    B[ix, iy] = 1.1*B[ix, iy]
    return
end

```

The smaller kernels are not run in a for loop in actual code. In the actual code the same kernel is called but with different set of arguments.

My guess is that it takes a lot more time to sync the kernel than running the it. But smaller kernels ( ASubset! and BSubset!) do not need to be synchronized. Is it possible to launch them without syncing?

This is a screenshot from the profiler which shows the problem.

 ![Screenshot from 2025-04-08 14-01-33](https://global.discourse-cdn.com/julialang/original/3X/a/8/a866f5e1d5260290135bdea6857dd0dcfc59cb86.png)

Alternatively, I might be completely wrong about what kills the performance and would be happy to know what is wrong and how to get more performance.

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [April 8, 2025, 1:30pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/2 "2025-04-08T13:30:37Z")

</div>

If I got your question right then this is the answer:

```julia
julia> using ParallelStencil

help?> @parallel_async
  @parallel_async kernelcall 
  @parallel_async ∇=... kernelcall

  │ Advanced
  │
  │ @parallel_async ranges kernelcall
  │ @parallel_async nblocks nthreads kernelcall
  │ @parallel_async ranges nblocks nthreads kernelcall
  │ @parallel_async (...) configcall=... backendkwargs... kernelcall
  │ @parallel_async ∇=... ad_mode=... ad_annotations=... (...) backendkwargs... kernelcall

  Declare the kernelcall parallel as with @parallel (see @parallel for more
  information); deactivates however automatic synchronization at the end of the call.
  Use @synchronize for synchronizing.

  │ Performance note
  │
  │ @parallel_async falls currently back to running synchronously if the
  │ package Threads or Polyester was selected with
  │ [`@init_parallel_stencil`](@ref).

  See also: @synchronize, @parallel

julia> 

```

Then you can synchronize them later with `@synchronize` (which can also synchronize a single stream).

---

<div class="post-metadata">

**Author:** ![pchaubal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pchaubal/32/213778_2.png) [@pchaubal](https://discourse.julialang.org/u/pchaubal)\
**Post date:** [April 8, 2025, 2:10pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/3 "2025-04-08T14:10:57Z")

</div>

Thanks for the answer. It partly addresses my question but doesn’t unlock much performance.

I replaced the call to second small kernel (BSubset!) with the async call as suggested.

```julia
        for i in 1:10:300
            @parallel_async (1:nx, i:i+10) BSubset!(B)
        end
        @synchronize

```

As seen from the profiler output, it does speedup things a little since the kernels are launched with less delay between two subsequent kernels. However the kernels are still launched serially rather than all at once.

 ![Screenshot from 2025-04-08 14-59-18](https://global.discourse-cdn.com/julialang/original/3X/c/7/c72bbe748b00583299d8af487eee8db652e308f8.png)

The smaller kernels all in total represent about ~30% of the computation of the big kernel. Yet, the the big kernel takes about 9 microseconds while the smaller kernels take about 80 microseconds. Is there a way to address this?

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [April 8, 2025, 2:44pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/4 "2025-04-08T14:44:20Z")

</div>

To run them all at once, you need to run them on different streams. You can pass the keyword argument `stream = ParallelStencil.ParallelKernel.@get_stream(i)` to `@parallel_async` where i is a stream index starting at 1. Then you can synchronize all the streams using `@synchronize ParallelStencil.ParallelKernel.@get_stream(i)`.

If these small kernels can also overlap with the large kernels, and you have also communication to hide then this can all automatically be done with the `@hide_communication` macro (see `?@hide_communication`). I guess one could add a macro to automatically overlap kernels in cases like yours (besides the one to hide communication and overlap boundary condition computations with inner point computations). However, it could typically be a better approach to create heavier kernels, computing also for example multiple batches within one kernel.

---

<div class="post-metadata">

**Author:** ![pchaubal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pchaubal/32/213778_2.png) [@pchaubal](https://discourse.julialang.org/u/pchaubal)\
**Post date:** [April 8, 2025, 4:32pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/5 "2025-04-08T16:32:12Z")

</div>

This seems to be the solution I am looking for. I tried launching the small kernels on different streams as suggested like `@parallel_async (1:nx, i:i+10) stream=stream BSubset!(B)`. They do show up on different streams in the profiler but they start one after the other making them effectively serial. So only a single stream is being run at any given time.

Why does this happen and any suggestions on how to avoid it?

---

<div class="post-metadata">

**Author:** ![samo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/samo/32/35398_2.png) [@samo](https://discourse.julialang.org/u/samo)\
**Post date:** [April 8, 2025, 4:50pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/6 "2025-04-08T16:50:34Z")

</div>

Check out the following: [CUDA streams do not overlap](https://discourse.julialang.org/t/cuda-streams-do-not-overlap/24879)

… and note that you can also use `ParallelStencil.ParallelKernel.@get_priority_stream(i)`.

However, you might rather want to create one or a few larger kernel instead of all these small kernels…

---

<div class="post-metadata">

**Author:** ![pchaubal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pchaubal/32/213778_2.png) [@pchaubal](https://discourse.julialang.org/u/pchaubal)\
**Post date:** [April 8, 2025, 5:31pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/7 "2025-04-08T17:31:05Z")

</div>

Yes, it seems that writing one bigger aggregated kernel seems a better choice. I wanted to avoid that as the code will become complex and unreadable.

Thanks for referring to the previous issue. I will try to run the code with priority stream to see if I can make my kernels overlap.

---

<div class="post-metadata">

**Author:** ![luraess](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luraess/32/16189_2.png) [@luraess](https://discourse.julialang.org/u/luraess)\
**Post date:** [April 9, 2025, 12:04pm UTC](https://discourse.julialang.org/t/with-parallelstencil-is-it-possible-to-launch-multiple-kernels-and-sync-later/127843/8 "2025-04-09T12:04:40Z")

</div>

If using CUDA.jl \> 5.4, it could be that some implicit synchronisation is occurring when accessing the same underlying GPU memory from different streams as discussed in [Ability to opt out of / improved automatic synchronization between tasks for shared array usage · Issue #2617 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/issues/2617). If this is the case, then one solution may be [Support disabling implicit synchronization by vchuravy · Pull Request #2662 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/pull/2662).