# CUDA global synchronization HOWTO

**URL:** <https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920>\
**Category:** Performance\
**Tags:** gpu, gpuarrays, cuda\
**Created:** [January 20, 2022, 9:57am UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920 "2022-01-20T09:57:17Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [January 20, 2022, 9:57am UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/1 "2022-01-20T09:57:17Z")

</div>

Dear Julianners,  
I try to create an algorithm that runs an elementwise update operation and a reduction in 10k iteration and about 1\_000\_000 times, so the kernel restarts(2-8us) are really expensive in this scenario.  
The algorithm is very simple but on GPU I need to sync all the calculations before the reduce\_sum.

I simplified the whole algorithm to this:

```julia
using BenchmarkTools
using CUDA
cudatestSync(c, a, b, X) = begin
	I = ((blockIdx().x - 1) * blockDim().x + threadIdx().x)
	I > size(c,1) * size(c,2) && return
	ca = CUDA.Const(a)
	cb = CUDA.Const(b)
	for i in 1:20
		c[I] += ca[I] + cb[I] 
	end
	# sync_threads()
	# CUDA.device_synchronize()
	# sync_threads()
	# device_synchronize()
	# while syncCounter<size(c,1) * size(c,2)
	# if ok[I]==0
	# atomic_inc!(syncCounter);
	# ok[I]+=1
	# end
	# end
	# nanosleep(1000_000_000)
	# threadfence()
	# threadfence_block()
	# synchronize()
	gh = this_grid()
	sync_grid(gh)
	if I <= size(c,1) 
		for j in 1:size(c,2)
			X[I] += c[I,j]
		end
	end
	return
end
nth=512	
N = 1000
cf,af,bf=CUDA.randn(Float32, N, N),CUDA.randn(Float32, N, N),CUDA.randn(Float32, N, N)
C,A,B = copy(cf),copy(af),copy(bf); X=CUDA.zeros(Float32,N)
@time CUDA.@sync @cuda threads=nth blocks=cld(N*N, nth) cooperative=true cudatestSync(C,A,B,X))
@show sum(cf .+ 20 .* (af .+ bf) ; dims=2)[1:10]
@show X[1:10]
@assert all(sum(cf .+ 20 .* (af .+ bf) ; dims=2) .≈ X) "sync doesn't WORK!"

```

I tried many different method that is supportted by the CUDA.jl. I know lots of them is just stupid try, I just let it there as I just tried them out.  
I believed cooperative groups would be the way to go:  
[`https://developer.nvidia.com/blog/cooperative-groups/`](https://developer.nvidia.com/blog/cooperative-groups/)  
[cuda-robus-scalable.pdf)](https://on-demand.gputechconf.com/gtc/2017/presentation/s7622-Kyrylo-perelygin-robust-and-scalable-cuda.pdf)  
But I get  
`ERROR: LoadError: CUDA error: too many blocks in cooperative launch (code 720, ERROR_COOPERATIVE_LAUNCH_TOO_LARGE)`

So the goal is to reach global synchronization before the `reduce_sum`. How can I do it without exiting a kernel?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 20, 2022, 10:00am UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/2 "2022-01-20T10:00:57Z")

</div>

Global synchronization, across SMs, is just not what CUDA is meant for. Cooperative groups can hide that, but it’ll impact performance and the possible launch configuration (as seen here). I recommend you try to fuse more so that you can reduce the number of launches without requiring a sync. If you really need a global sync, only launch a single block so that `syncthreads` is global.

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [January 20, 2022, 11:01am UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/3 "2022-01-20T11:01:21Z")

</div>

I fused 3 kernel call into one, the only last one remained the `reduce_sum`, but If I could fuse this I could reduce the kernel calls by 2x 10\_000 times as I could move the whole cycle into the kernel.

For me it is extremly big deal and important to be able to fuse this. This algorithm could be so ridiculous fast with that, I cannot even imagine. Now it needs like 25 hours to run.  
I have to do it only once. Any possible solution would be great!

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [January 20, 2022, 11:44am UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/4 "2022-01-20T11:44:34Z")

</div>

The way they did it sounds extremely elegant:

> **[Global thread barrier](https://forums.developer.nvidia.com/t/global-thread-barrier/8498/58)**
>
> peastman, andradx I think I found good and fast solution of your iteration problem. global blocknum=0; global completedblockquantity=0; at the start of the block shared current\_block\_num; shared int currentiteration, blockindex; if...

Is this possible?

For me I cannot make atomic\_inc!(block\_num) work due to:

```julia
KernelError: kernel returns a value of type `Union{}`

Make sure your kernel function ends in `return`, `return nothing` or `nothing`.
If the returned value is of type `Union{}`, your Julia code probably throws an exception.
Inspect the code with `@device_code_warntype` for more details.

```

What can be the problem?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 20, 2022, 11:46am UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/5 "2022-01-20T11:46:33Z")

</div>

You’re invoking this method incorrectly. Please read the documentation: [Troubleshooting · CUDA.jl](https://cuda.juliagpu.org/stable/development/troubleshooting/#KernelError:-kernel-returns-a-value-of-type-Union%7B%7D). Look at the tests for valid invocations of `atomic_inc!`.

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [January 20, 2022, 12:07pm UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/6 "2022-01-20T12:07:14Z")

</div>

Ok, I was calling a wrong function all the time. XD Great… atomic\_inc! works as described. Btw… the way it works is mindbogling. 😃 I was like it will be like a `++i`

Totally excited about this method they described. :o

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 20, 2022, 12:12pm UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/7 "2022-01-20T12:12:55Z")

</div>

That only works for your particular kernel/set-up/inputs. In general it is impossible, because you can have a configuration where not all blocks can be scheduled at the same time, due to resource constraints. If you then come up with some sort of operation that requires all threads to reach that point, you’ll just deadlock.

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [January 20, 2022, 12:48pm UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/8 "2022-01-20T12:48:12Z")

</div>

Yeah… that is a very serious problem.  
I should ask “the thread to stop at this point and work on another thread’s job and reschedule when condition succeed”. ☹

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 20, 2022, 12:50pm UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/9 "2022-01-20T12:50:18Z")

</div>

That’s not possible with the current programming model. Hence; “you shouldn’t be doing this”. The thread you linked to has several other people proclaiming the same thing. If it works for your specific problem at hand, that’s great, but don’t expect it to generalize to other problem sizes / hardware / Julia versions / data types / … without carefully verifying it does.

---

<div class="post-metadata">

**Author:** ![Marcell\_Havlik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/marcell_havlik/32/37424_2.png) [@Marcell\_Havlik](https://discourse.julialang.org/u/Marcell_Havlik)\
**Post date:** [January 20, 2022, 12:52pm UTC](https://discourse.julialang.org/t/cuda-global-synchronization-howto/74920/10 "2022-01-20T12:52:47Z")

</div>

No, you should be right in my scenario too. The dataset is bigger than the ‘parallel lanes’ or how to say.

Shit I misscalculated this very much. I was calculating with finishing “threads”. But this should make a deadlock.

This is VERY sad. I really wanted to spare the kernel starts. ☹