@inbounds slower

eldee
Thu, 06 Mar 2025 13:27:35 +0000
GPU

Hi,

I’m encountering some situations in my GPU code where adding @inbounds increases execution time by 10%-20%. Of course it would be great if we would never need @inbounds, but on the other hand there are certainly also situations where adding it does help considerably.

Is there any theoretical reason why using @inbounds might negatively affect performance? Performing less work by omitting checks sounds like it should normally never harm performance, but perhaps there are some subtle incompatibilities, similar to the AVX512 / SIMD CPU situation here: @inbounds code slower than one without? How can one predict in advance if it will be useful to add @inbounds?

I struggle with extracting a good MWE. In the code below, the difference in speed is small, but at least it is quite consistent. Hopefully any conclusions will still carry over to the full code.

using CUDA, Random, Statistics
using CUDA: i32

function main()
    inbounds_times = Float64[]
    noinbounds_times = Float64[]
    for i = 1:1001  # Takes around 1.5 minutes on my RTX 3070
        P = CUDA.rand(3, 16, 2_000_000)
        G = similar(P)
        M = similar(P)
        I = CuArray{Int32}(sort(@view randperm(size(P, 3))[1:500_000]))
        # We will use the equivalent of @view P[:,:,I].

        # Just in case there's some caching going on (though I doubt it),
        # alternate the order of inbounds and noinbounds for a fair comparison
        if i % 2 == 0
            inbounds_time = @CUDA.elapsed @cuda threads=512 blocks=cld(length(I), 512) inbounds_kernel!(P, G, M, I)
            noinbounds_time = @CUDA.elapsed @cuda threads=512 blocks=cld(length(I), 512) noinbounds_kernel!(P, G, M, I)
        else
            noinbounds_time = @CUDA.elapsed @cuda threads=512 blocks=cld(length(I), 512) noinbounds_kernel!(P, G, M, I)
            inbounds_time = @CUDA.elapsed @cuda threads=512 blocks=cld(length(I), 512) inbounds_kernel!(P, G, M, I)
        end

        # Don't include the first runs
        if i >= 2
            push!(inbounds_times, inbounds_time)
            push!(noinbounds_times, noinbounds_time)
        end

        CUDA.unsafe_free!(P)
        CUDA.unsafe_free!(G)
        CUDA.unsafe_free!(M)
        CUDA.unsafe_free!(I)
    end
    println("Inbounds: ")
    println("\tMin: ", minimum(inbounds_times) * 1000)
    println("\tMedian: ", median(inbounds_times) * 1000)
    println("\tMean: ", mean(inbounds_times) * 1000)
    println()
    println("No inbounds: ")
    println("\tMin: ", minimum(noinbounds_times) * 1000)
    println("\tMedian: ", median(noinbounds_times) * 1000)
    println("\tMean: ", mean(noinbounds_times) * 1000)
end

function inbounds_kernel!(P, G, M, I)
    idx = threadIdx().x + (blockIdx().x - 1i32) * blockDim().x
    @inbounds while idx <= length(I)
        i = I[idx]

        for j = axes(P, 1)      # (I guess the compiler might automatically eliminate bounds-checking for P.)
            for k = axes(P, 2)  # (But explicitly using 1i32:3i32 and 1i32:16i32 yields similar timings.)

                g = G[k, j, i]    # (no memory coalescing)
                m = M[k, j, i]

                m = 0.9 * m + 0.1 * g

                M[k, j, i] = m
                P[k, j, i] -= m
            end
        end

        idx += blockDim().x * gridDim().x
    end
    return
end

function noinbounds_kernel!(P, G, M, I)
    idx = threadIdx().x + (blockIdx().x - 1i32) * blockDim().x
    while idx <= length(I)  # not @inbounds
        i = I[idx]

        for j = axes(P, 1)
            for k = axes(P, 2)
                g = G[k, j, i]
                m = M[k, j, i]

                m = 0.9 * m + 0.1 * g

                M[k, j, i] = m
                P[k, j, i] -= m
            end
        end

        idx += blockDim().x * gridDim().x
    end
    return
end

main()

Inbounds:
        Min: 25.07782354950905
        Median: 25.257023982703686
        Mean: 25.382457310333848

No inbounds:
        Min: 24.692736566066742
        Median: 24.902560748159885
        Mean: 25.018115950748324

versioninfo (click for more details)
9 posts - 3 participants