# GPU kernel does not scale properly with data
**URL:** https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863
**Category:** GPU
**Tags:** question
**Created:** [December 15, 2024, 2:46pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863 "2024-12-15T14:46:45Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Giorgos\_Vretinaris](https://discourse.julialang.org/u/Giorgos_Vretinaris)
#### Post date: [December 15, 2024, 2:46pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/1 "2024-12-15T14:46:45Z")
Hello, I am trying to write a ODE solver on the GPU. I am solving the same simple ODE but for different initial data. I tried the `EnsembleGPUArray` option from [DiffEqGPU.jl](https://juliahub.com/ui/Packages/General/DiffEqGPU) but it was quite slow.
The kernel I wrote is reading three `CellArray` objects, where one contains the state vectors, so the cell is of type `SVector{5,Float32}`, the other one is just a register for the Euler step, so a copy of the first one, and the last one contains parameters and is of type `SVector{3,Float32}`. Using these objects, it performs only one Euler step. The RHS of my ODE for one state vector on the CPU takes about 70ns.
The effective bandwidth on my Tesla K40c goes like:
The saxpy kernel
```julia
function _saxpy_kernel!(a, x, y)
i = (blockIdx().x - 1) * blockDim().x + threadIdx().x
if i <= length(y)
@inbounds y[i] = a * x[i] + y[i]
end
return nothing
end
```
gives a Teff≈ 178 GB/s. So the ODE kernel is much slower.
Could you please help me understand what I’m doing, or even thinking, wrong?
```julia
function evolve_gpu_kernel!(photons,photon_params,RHS,time_stepper,params)
N,M,tmps,dt = params
ix = (blockIdx().x - 1) * blockDim().x + threadIdx().x
iy = (blockIdx().y - 1) * blockDim().y + threadIdx().y
if ix <= N && iy <= M
photons[ix,iy] = time_stepper(RHS,tmps[ix,iy],photons[ix,iy],photon_params[ix,iy],dt)
end
return
end
function Euler(rhs,du,u,p,dt)
du = rhs(u,p)
return u .+ du .* dt
end
function run(p,nthreads=nothing)
metric,N,M = p
θ0 = deg2rad(87.5)
alims = (-9,9) ### pθ/pt impact param
blims = (-10,10) ### -pφ/pt impact param
pos = SVector{D,T}([0,init_r,θ0,π/2])
inputs,ps = generate_rays_gpu(pos, metric, N, M, alims, blims)
photons_tmp = Matrix(reduce(hcat,vec(inputs))') |> cu
photon_params_tmp = Matrix(reduce(hcat,vec(ps))') |> cu
photons = CuCellArray{SVector{5,Float32}}(undef,N,M)
tmp = CuCellArray{SVector{5,Float32}}(undef,N,M)
photon_params = CuCellArray{SVector{3,Float32}}(undef,N,M)
photons.data .= photons_tmp
photon_params.data .= photon_params_tmp
CUDA.unsafe_free!(photons_tmp)
photons_tmp = nothing
CUDA.unsafe_free!(photon_params_tmp)
photon_params_tmp = nothing
RHS = kerr_geod
time_stepper = Euler
dt = 0.01f0
if isnothing(nthreads)
MAX_THREADS_PER_BLOCK = CUDA.attribute(device(), CUDA.DEVICE_ATTRIBUTE_MAX_THREADS_PER_BLOCK)
nthreads = (isqrt(MAX_THREADS_PER_BLOCK), isqrt(MAX_THREADS_PER_BLOCK))
end
nblocks = cld.((N,M),nthreads)
params = (N,M,tmp,dt)
t = CUDA.@elapsed CUDA.@sync @cuda(threads=nthreads,blocks=nblocks,
evolve_gpu_kernel!(photons,photon_params,RHS,time_stepper,params))
return t
end
function plot_diagnostic()
idxs = [i for i in 4:10]
x = 2 .^ idxs
labels = [latexstring("2^{$idx}") for idx in idxs]
time = n -> run((kerr_metric_bl,n,n),(16,16));
t = time.(x)
Aeff(N) = (2*5+3)*1e-9*N^2*sizeof(Float32) ## GB
Teff = @. Aeff(x)/t
plot(x,t .* 1e3,
xlabel="N", ylabel="Wall Time [ms]",
label=nothing, tickfontsize = 14, dpi=144,
xticks=(x,labels),xaxis=:log2)
scatter!(x,t .* 1e3,label=nothing)
savefig("wall_times.png")
plot(x,Teff,
xlabel="N", ylabel=L"$T_{\mathrm{eff}}$ [GB/s]",
label=nothing,
xticks=(x,labels),xaxis=:log2)
scatter!(x,Teff,label=nothing)
savefig("Teff.png")
end
```
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [December 16, 2024, 2:06pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/2 "2024-12-16T14:06:27Z")
Did you forget `@inbounds`? It’s possible that generates a lot of extra code, lowering occupancy. Try running under `CUDA.@profile trace=true` for details (notably, comparing the launch configuration and register use of SAXPY vs your kernel).
---
### Author:  [@Giorgos\_Vretinaris](https://discourse.julialang.org/u/Giorgos_Vretinaris)
#### Post date: [December 16, 2024, 3:02pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/3 "2024-12-16T15:02:47Z")
Thank you for your reply. I get the following error when running `CUDA.@profile trace=true`:
```julia
[ Info: This Julia session is already being profiled; defaulting to the external profiler.
ERROR: MethodError: no method matching profile_externally(::var"##132#profiled_code"; trace::Bool)
Closest candidates are:
profile_externally(::Any) got unsupported keyword argument "trace"
@ CUDA ~/.julia/packages/CUDA/Tl08O/src/profile.jl:130
```
The server that I am using does not have the external profilers shown in the docs of CUDA.jl, should I contact the admin to have them installed or is there any way to deal with this without the external profilers?
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [December 16, 2024, 4:35pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/4 "2024-12-16T16:35:47Z")
You shouldn’t have to install any external profiler for that functionality, in fact, I think you may be misinterpreting the error message: You’re somehow already running Julia already under a profiler, as attaching a new profiler fails (with an error that’s hard to interpret differently): [CUDA.jl/src/profile.jl at 4e9513b8a4e56629a236b58504d609b1775a8236 · JuliaGPU/CUDA.jl · GitHub](https://github.com/JuliaGPU/CUDA.jl/blob/4e9513b8a4e56629a236b58504d609b1775a8236/src/profile.jl#L144-L162)
---
### Author:  [@Giorgos\_Vretinaris](https://discourse.julialang.org/u/Giorgos_Vretinaris)
#### Post date: [December 16, 2024, 4:48pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/5 "2024-12-16T16:48:41Z")
I’m sorry but I am not exactly following you, as I am lacking a lot of knowledge, probably in Julia itself too.
I am not loading any profilers, at least not explicitly, I am only loading the following
```julia
using CellArrays, CUDA
using LinearAlgebra
using StaticArrays
using LaTeXStrings
using Images
using Plots
```
And this error pops up when I run the kernel under `CUDA.@profile trace = true`.
The error is not there if I don’t include the `trace = true` part, but then I just get a `CUDA.KernelState` object.
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [December 16, 2024, 7:31pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/6 "2024-12-16T19:31:14Z")
How are you starting `julia`?
---
### Author:  [@Giorgos\_Vretinaris](https://discourse.julialang.org/u/Giorgos_Vretinaris)
#### Post date: [December 16, 2024, 7:40pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/7 "2024-12-16T19:40:59Z")
I run `julia -t 32` and I am currently using version 1.10.3.
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [December 16, 2024, 8:01pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/8 "2024-12-16T20:01:05Z")
Are you perhaps nesting `CUDA.@profile` invocations?
---
### Author:  [@Giorgos\_Vretinaris](https://discourse.julialang.org/u/Giorgos_Vretinaris)
#### Post date: [December 16, 2024, 8:04pm UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/9 "2024-12-16T20:04:51Z")
I only run
```julia
CUDA.@profile trace = true CUDA.@sync @cuda(threads=nthreads,blocks=nblocks,
evolve_gpu_kernel!(photons,photon_params,RHS,time_stepper,params))
```
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [December 17, 2024, 6:06am UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/10 "2024-12-17T06:06:34Z")
That’s bizarre. What happens if you run [the logic of the CUPTI detection](https://github.com/JuliaGPU/CUDA.jl/blob/4e9513b8a4e56629a236b58504d609b1775a8236/src/profile.jl#L151-L154) in isolation? Does it throw the same error?
---
### Author:  [@Giorgos\_Vretinaris](https://discourse.julialang.org/u/Giorgos_Vretinaris)
#### Post date: [December 17, 2024, 8:40am UTC](https://discourse.julialang.org/t/gpu-kernel-does-not-scale-properly-with-data/123863/11 "2024-12-17T08:40:56Z")
I can’t seem to find a way to call it via CUDA, so by copying and pasting the function onto the REPL after including the packages above, yields `true` for the entire `detect_cupti` function but the following snippet:
```julia
cfg = CUPTI.ActivityConfig([])
CUPTI.enable!(cfg) do
# do nothing
end
```
returns
```julia
ERROR: CUPTIError: CUPTI doesn't allow multiple callback subscribers. Only a single subscriber can be registered at a time. (code 39, CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED)
```