Hi, I’m trying to use different GPU kernels for overlapping operations because a single kernel usually cannot fully utilize GPU hardware resources. I defined a kernel function and tried the following, but the running time did not shorten as I expected, but it was almost equal to the sum of the seri…

[image] fred_wu: does the CUDA driver randomly determine it, and the programmer cannot control it? It’s not random, of course, but no you do not have control over it. You can only express which tasks can execute concurrently, but the driver will decide (based on resource availability and kern…

Thanks for your reply. I used NSight Systems to analyze and found that the timelines of the two functions did not overlap. I want to ask again, under what circumstances will this programming method have an acceleration effect, or does the CUDA driver randomly determine it, and the programmer cannot…

Thanks for your reply. It helped me a lot. I wrote the following code to try again, but what puzzles me is why the running time of the overlap operation is longer and there are more allocations? My previous understanding is that the running time of the overlap operation should be less than or equal…

You should use an actual profiler, as I suggested before. Then you would see that your kernels are so short that overlap isn’t possible (as they finish before the next kernel is queued): [image] Increasing the problem size here doesn’t “help”, because then the GPU seems fully occupied performing …

Thank you for your thorough response! I comprehend.

And FYI, in the next CUDA.jl version you’ll be able to use the integrated profiler to figure this out too (albeit not in a nice graphical way): julia> CUDA.@profile trace=true @sync begin @async begin α = CUDA.dot(a,b) end @async begin …

How to perform GPU overlap operations on the custom kernel function?

Specific Domains GPU

maleadt August 16, 2023, 10:32am 2

Your program is written correctly, whether execution will overlap or not is up to the CUDA driver. You also better use a proper profiler to verify whether execution overlaps or not (i.e., NSight Systems).

Topic		Replies	Views
CUDA streams do not overlap GPU question	6	3161	July 1, 2019
Questions about using CUDA.jl for GPU concurrent programming: Computational results cannot be obtained when overlapping GPU and CPU operations GPU question	2	463	April 12, 2023
Can the CPU function be a multi-process parallel function when using the Threads.@spawn command to perform overlapping operations between the GPU and the CPU? GPU question , concurrency	3	283	August 24, 2023
Asynchronous kernel scheduling with KernelAbstractions GPU	5	413	June 25, 2023
Is is possible to merge multiple kernels in CUDAnative to minimize launch overhead and execution overhead? GPU	12	1687	November 11, 2018

How to perform GPU overlap operations on the custom kernel function?

Related topics