# Synchronize streams in CUDA.jl

**URL:** <https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127>\
**Category:** GPU\
**Tags:** gpu, cuda\
**Created:** [August 13, 2024, 10:14am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127 "2024-08-13T10:14:15Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 13, 2024, 10:14am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/1 "2024-08-13T10:14:15Z")

</div>

Which of the synchronize commands is supposed to be used to synchronize the streams in CUDA? I am able to synchronize them with `device_synchronize()`, but this seems to have a huge impact on speed. Is there perhaps a more ‘lightweight’ command that could be used?

---

<div class="post-metadata">

**Author:** ![luraess](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/luraess/32/16189_2.png) [@luraess](https://discourse.julialang.org/u/luraess)\
**Post date:** [August 13, 2024, 10:36am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/2 "2024-08-13T10:36:02Z")

</div>

`CUDA.synchronize()` syncs streams. Also, unless you need to access data across different streams (non-default ones), time kernel execution, and some other specific operations (mostly across tasks and streams), kernel execution is stream-ordered, meaning you may skip explicit stream sync.

---

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 13, 2024, 11:13am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/3 "2024-08-13T11:13:16Z")

</div>

Thanks for the quick reply! I might be using streams in a slightly unconventional way. I’m launching multiple kernels asynchronously using streams (1 kernel launch per stream), and then, performing a single large DtoH memcopy.

I notice that the memcopy, which uses the default stream, starts before all streams complete kernel execution. This obviously causes incorrect results. I want to ensure all streams complete kernel execution before the memcopy. `CUDA.synchronize` did not seem like it was doing this.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [August 13, 2024, 12:40pm UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/4 "2024-08-13T12:40:34Z")

</div>

> [@Cibin\_Joseph](#):
>
> I notice that the memcopy, which uses the default stream, starts before all streams complete kernel execution.

We don’t use the traditional default stream. Instead, CUDA.jl allocates a default stream per task, so one possible solution here is to launch the threads on different Julia tasks (using `@async`), call `synchronize()` at the end of each task, and `wait` or `@sync` (the Base one) on those tasks. After that, you should be safe to perform the memory copy, without ever having to perform a blocking, let alone device-wide synchronization.

There’s alternatives, like using `CuEvent`s to make the streams depend on one another, but the above one is probably the most idiomatic.

Note that if you’re working on a single array per stream, CUDA.jl will be able to automatically figure out the need for synchronization on another stream, see [CUDA.jl 5.4: Memory management mayhem ⋅ JuliaGPU](https://juliagpu.org/post/2024-05-28-cuda_5.4/#tracked_memory_allocations), “Using multiple streams”.

---

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 14, 2024, 6:32pm UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/5 "2024-08-14T18:32:44Z")

</div>

That’s interesting!  
Why do you suggest calling synchronize at the end of each task? Wouldn’t the @sync that wraps the block, synchronize all threads/streams when they finish execution?

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [August 14, 2024, 7:28pm UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/6 "2024-08-14T19:28:28Z")

</div>

`@sync != CUDA.@sync`

---

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 21, 2024, 9:59pm UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/7 "2024-08-21T21:59:23Z")

</div>

Thanks @maleadt and @carstenbauer for your tips! I was able to make some progress.

Unfortunately I’m not working with a single array per stream and now am trying to figure out how to copy a matrix slice over a stream. I saw that `unsafe_copy2d!` was my only option if I wanted to specify a stream in the arguments, but haven’t got it to work correctly yet.

Should I use that to copy an array slice from device to host or is there a better option?

Here’s an MWE to clarify what I’m trying to achieve:

```julia
using CUDA

function kernel!(a, i)
   # A large computation
    m::Int = 10^4
    for j = 1:m 
        for k = 1:m
            a[threadIdx().x, i] = i
        end
    end
    return
end

function main()
    n = 2 # No. of streams

    a = rand(256, n)
    a_d = CuArray(a)

    @sync for i in 1:n
        @async begin
            stream = CuStream()
            
            # Each stream handles a column of a_d
            @cuda threads=256 stream=stream kernel!(a_d, i)
            
            # Copy each column of a_d to a
            # THIS NEEDS TO BE DONE USING THE CURRENT STREAM
            a[:, i] .= Array(a_d[:, i])
        end
    end 
end 

CUDA.@profile main()

```

From the profiler output, the memcopy occurs on different streams from the kernel launch and aren’t synchronized with them. I’d like them to occur on the same stream after the kernel operation is completed.

 ![image](https://global.discourse-cdn.com/julialang/original/3X/2/a/2a29035931b18c1ab1345cff1b47c4d557b62eaa.png)

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [August 22, 2024, 9:31am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/8 "2024-08-22T09:31:04Z")

</div>

> [@Cibin\_Joseph](#):
>
> figure out how to copy a matrix slice over a stream

What do you mean by that, specifically? If you just want the copy on another stream, you can use `CUDA.stream!` with do-block syntax to temporarily switch the current task to another stream, and perform a regular copy within. That’s even supposed to work when the kernels producing data on the array haven’t finished yet, as CUDA.jl should synchronize the previous stream that your array was used on when performing the copy and detecting a different stream, but I haven’t tested that specifically.

---

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 22, 2024, 9:50am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/9 "2024-08-22T09:50:11Z")

</div>

Sorry if that wasn’t clear.  
What I meant was, in the example above, when I do

```julia
@sync for i = 1:n
  @async begin
    stream=CuStream()
    @cuda threads=256 stream=stream kernel!(a_d, i)

    a[:, i] .= Array(a_d[:, i])
  end
end

```

for each `i`, the kernel launch and the memcopy occur on two different streams.

I want the memcopy to occur on the same stream _after_ the kernel computation on that stream is completed.

I didn’t know about the `do ... end` block with streams though! That looks promising.

---

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 22, 2024, 9:53am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/10 "2024-08-22T09:53:40Z")

</div>

> That’s even supposed to work when the kernels producing data on the array haven’t finished yet, as [CUDA.jl](https://juliahub.com/ui/Packages/General/CUDA) should synchronize the previous stream that your array was used on when performing the copy and detecting a different stream, but I haven’t tested that specifically.

I wonder if this functionality isn’t working for me because all streams are accessing the same array. I don’t have a single array per stream.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [August 22, 2024, 11:28am UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/11 "2024-08-22T11:28:26Z")

</div>

> [@Cibin\_Joseph](#):
>
> ```julia
> stream=CuStream()
> @cuda threads=256 stream=stream kernel!(a_d, i)
> 
> ```

I wouldn’t do that. It’s possible, but just change the stream for that task / `@async` block by calling `stream!` (without the do-block syntax).

> [@Cibin\_Joseph](#):
>
> I don’t have a single array per stream.

Yeah that could be problematic, as it will cause additional synchronization. The assumption is currently baked into CUDA.jl’s memory handling: [CUDA.jl/src/memory.jl at 76e2972814a0e7910f35ed3ad17b1a9198628f34 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/blob/76e2972814a0e7910f35ed3ad17b1a9198628f34/src/memory.jl#L565-L569)

What’s the reason behind sharing arrays between different streams? Do you really have several kernels operating on subsets of the memory, only to copy the pieces of memory that are ready?

---

<div class="post-metadata">

**Author:** ![Cibin\_Joseph](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cibin_joseph/32/208886_2.png) [@Cibin\_Joseph](https://discourse.julialang.org/u/Cibin_Joseph)\
**Post date:** [August 23, 2024, 10:58pm UTC](https://discourse.julialang.org/t/synchronize-streams-in-cuda-jl/118127/12 "2024-08-23T22:58:47Z")

</div>

The reason for sharing a single array between different streams was to avoid redundant data copying from HtoD.

I’m working on a particle interaction problem where there are certain common clusters of particles that interact with ‘target’ particles. I was initially using separate arrays per stream, but then, those common clusters were copied over to the GPU redundantly.  
When each stream works on a subset of the large array, there are no redundant copy operations, and instead just a single large initial copy operation.

At this point, these are experimental algorithms, so I’m still figuring out if there’s a better way to do it, but for now, the `stream! do ... end` block seems to do the job for me like so:

```julia
streams = Vector{CuStream}(undef, nstreams)

# Launch all streams asynchronously
for i = 1:nstreams
    streams[i] = CuStream()
    @cuda threads=256 stream=streams[i] kernel!(a_d, i)
end

# Memcopy using the same streams to ensure synchronization
for i = 1:nstreams
    stream!(streams[i]) do
        a[:, i] .= Array(a_d[:, i])
    end
end

```

However, like you mentioned, I think there might be additional synchronization that’s occurring that’s costing me performance.