# Some CUDA functions suddenly become very slow

**URL:** <https://discourse.julialang.org/t/some-cuda-functions-suddenly-become-very-slow/117025>\
**Category:** New to Julia\
**Created:** [July 14, 2024, 9:32am UTC](https://discourse.julialang.org/t/some-cuda-functions-suddenly-become-very-slow/117025 "2024-07-14T09:32:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![baizhan](https://avatars.discourse-cdn.com/v4/letter/b/ecae2f/32.png) [@baizhan](https://discourse.julialang.org/u/baizhan)\
**Post date:** [July 14, 2024, 9:32am UTC](https://discourse.julialang.org/t/some-cuda-functions-suddenly-become-very-slow/117025/1 "2024-07-14T09:32:24Z")

</div>

I am new to GPU computation and I want to start from julia.  
I use `CUDA.jl` to do this and write some code following the instruction.

However, it takes very long time for the following simple code:

```julia
module TestMyCUDA
function test()
  CUDA.memory_status()

  num_c=100
  num_x=512
  @time array1 = CuArray{Float32}(rand(Float64,num_c,3))
  @time array2 = CUDA.zeros(Complex{Float32}, num_x,num_x,num_x) # allocate the lattice on the GPU

  @time @cuda dosomething!(array2,array1,num_x)
  return nothing
end

function dosomething!(array2,array1,num_x)
  index = (blockIdx().x - 1) * blockDim().x + threadIdx().x
  stride = gridDim().x * blockDim().x

  for N=index:stride:num_x^3

    i = (N-1) ÷ num_x^2 +1
    j = ((N-1) % num_x^2 ) ÷ num_x + 1
    k = ((N-1) % num_x^2 ) % num_x + 1

    array2[i,j,k]=1.0+1.0im
  end
  return nothing
end
end

import .TestMyCUDA
TestMyCUDA.test()

```

In this example code, I simply do 2 things: allocating two `CuArray`’s, and giving `array2` some value.

It works out when running it for the first time:

```julia
Effective GPU memory usage: 19.83% (3.172 GiB/16.000 GiB)
Memory pool usage: 2.000 GiB (2.031 GiB reserved)
  0.000130 seconds (14 allocations: 4.141 KiB)
  0.014412 seconds (851 allocations: 33.094 KiB, 82.98% compilation time)
  0.371885 seconds (86.76 k allocations: 7.021 MiB, 4.08% gc time, 19.74% compilation time)

```

However, when running it for second time, the first allocation will take a surprisingly long time:

```julia
Effective GPU memory usage: 26.08% (4.172 GiB/16.000 GiB)
Memory pool usage: 2.000 GiB (3.031 GiB reserved)
 63.046641 seconds (15 allocations: 4.156 KiB)
  0.000133 seconds (42 allocations: 1.766 KiB)
  0.579747 seconds (67.56 k allocations: 5.685 MiB, 4.38% compilation time)

```

Interestingly, it is not always the allocation that takes very long time.  
If I insert some other code, the `dosomething!` function will become very slow and the allocation function becomes normal speed.

Do anyone know why this happens?

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 14, 2024, 10:29am UTC](https://discourse.julialang.org/t/some-cuda-functions-suddenly-become-very-slow/117025/2 "2024-07-14T10:29:03Z")

</div>

CUDA APIs are asynchronous, so your initial timing is just the time to launch the kernel. The kernel actually does take 60s to finish, because its implemented badly: You’re iterating elements on a single GPU thread, which is the wrong way to use GPUs. Please read the CUDA.jl introductory tutorial, this exact pitfall is explained there: [Introduction · CUDA.jl](https://cuda.juliagpu.org/stable/tutorials/introduction/#Writing-your-first-GPU-kernel)

---

<div class="post-metadata">

**Author:** ![baizhan](https://avatars.discourse-cdn.com/v4/letter/b/ecae2f/32.png) [@baizhan](https://discourse.julialang.org/u/baizhan)\
**Post date:** [July 14, 2024, 11:22am UTC](https://discourse.julialang.org/t/some-cuda-functions-suddenly-become-very-slow/117025/3 "2024-07-14T11:22:35Z")

</div>

Thank you!

I think you are right that the “delay” comes from the asynchronous of CUDA.

Actually, in my real code, I am not using only one GPU thread.  
But my real code is also very heavy even for a lot of threads.

I made a mistake by simply use `@time` to find the bottleneck.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [July 14, 2024, 8:00pm UTC](https://discourse.julialang.org/t/some-cuda-functions-suddenly-become-very-slow/117025/4 "2024-07-14T20:00:23Z")

</div>

You can use `CUDA.@time` instead, which does the synchronization for you, or just make it a habit to add `CUDA.@sync` to time measurement utilities (it be `@time`, `@btime`, `@benchmark`, etc). Also consider using `CUDA.@profile` which would have showed you the actual execution time of your kernel.