# Improving cache behavior with ldg()

**URL:** <https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628>\
**Category:** GPU\
**Tags:** cudanative\
**Created:** [January 21, 2020, 4:27pm UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628 "2020-01-21T16:27:12Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michael\_Benton](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@Michael\_Benton](https://discourse.julialang.org/u/Michael_Benton)\
**Post date:** [January 21, 2020, 4:27pm UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/1 "2020-01-21T16:27:12Z")

</div>

What exactly is meant by [https://juliagpu.github.io/CUDAnative.jl/stable/lib/device/array.html](https://juliagpu.github.io/CUDAnative.jl/stable/lib/device/array.html)

that ldg “loads the value through the read-only texture cache for improved cache behavior” ?

The following is a memory bandwidth bound kernel that spends most of the execution time waiting on reads from global memory. It performs the same regardless of syntax ` ldg(in,I)`, or `in[I]`

```julia
using CuArrays, CUDAnative, BenchmarkTools
a = 10000;
b = 5000;
in = CuArrays.rand(a,b);
out = CuArrays.zeros(b);
function sum!(out,in)
  a = size(in,1)
  i = threadIdx().x + (blockIdx().x-1)*blockDim().x
  I = (i-Int32(1))*Int32(a)
  val = Float32(0)
  for j = 1:a
    I += 1
    @inbounds val += ldg(in,I)
    # @inbounds val += in[I]
  end
  @inbounds out[i] = val
  return nothing
end
@btime CuArrays.@sync @cuda threads=500 blocks=Int64(b/500) sum!(out,in)

```

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 21, 2020, 4:36pm UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/2 "2020-01-21T16:36:38Z")

</div>

Google for `__ldg`, it’s the same construct as from CUDA C.

---

<div class="post-metadata">

**Author:** ![Gnimuc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gnimuc/32/2194_2.png) [@Gnimuc](https://discourse.julialang.org/u/Gnimuc)\
**Post date:** [January 22, 2020, 4:49am UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/3 "2020-01-22T04:49:20Z")

</div>

> [@Michael\_Benton](#):
>
> What exactly is meant by [https://juliagpu.github.io/CUDAnative.jl/stable/lib/device/array.html](https://juliagpu.github.io/CUDAnative.jl/stable/lib/device/array.html)
> 
> that ldg “loads the value through the read-only texture cache for improved cache behavior” ?

It’s the same as using ` __restrict__ ` identifier in CUDA C/C++. You could read more about this topic in [this post](https://devblogs.nvidia.com/cuda-pro-tip-optimize-pointer-aliasing/).

> [@Michael\_Benton](#):
>
> It performs the same regardless of syntax ` ldg(in,I)` , or `in[I]`

It makes no difference because the global memory accessing pattern in your kernel is non-coalescing, and `in` is rarely hit in L1 cache.

---

<div class="post-metadata">

**Author:** ![Gnimuc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gnimuc/32/2194_2.png) [@Gnimuc](https://discourse.julialang.org/u/Gnimuc)\
**Post date:** [January 22, 2020, 5:04am UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/4 "2020-01-22T05:04:34Z")

</div>

The block size(`threads=500`) also looks wired to me, CUDA blocks are split up into warps(32 threads), there are still 16 warps running concurrently per block even if you set the block size to 500.

---

<div class="post-metadata">

**Author:** ![maleadt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/maleadt/32/10097_2.png) [@maleadt](https://discourse.julialang.org/u/maleadt)\
**Post date:** [January 22, 2020, 6:57am UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/5 "2020-01-22T06:57:47Z")

</div>

> [@Gnimuc](#):
>
> It’s the same as using ` __restrict__ ` identifier in CUDA C/C++. You could read more about this topic in [this post](https://devblogs.nvidia.com/cuda-pro-tip-optimize-pointer-aliasing/).

They are not really related: `restrict` teaches the compiler that pointers do not alias, which (often in combination with `const`) _might_ have it understand certain reads are from read-only memory and could be optimized as such. `ldg` bypasses any analysis and forces the hardware to read through the texture cache, which should only be used in the case of read-only memory. Using the latter with read-write memory _will_ result in corruption, while you can safely tag such pointers `restrict`.

> [@Gnimuc](#):
>
> It makes no difference because the global memory accessing pattern in your kernel is non-coalescing, and `in` is rarely hit in L1 cache.

Do you have a source for that? IIUC these reads are never cached in L1, but in L2 or in the read-only data cache depending on the use of `ldg`. I’m also not sure what memory coalescing has to do with that.

---

<div class="post-metadata">

**Author:** ![Gnimuc](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gnimuc/32/2194_2.png) [@Gnimuc](https://discourse.julialang.org/u/Gnimuc)\
**Post date:** [January 22, 2020, 11:16am UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/6 "2020-01-22T11:16:40Z")

</div>

Maybe I shouldn’t say “It’s the same”, as explained above, from a compiler’s perspective, they’re quite different. But what I really meant is that, from a user’s perspective, ` __restrict__ ` and `ldg` are just two alternative ways to enable read-only loads(ref: [S0514-GTC2012-GPU-Performance-Analysis page 32](http://on-demand.gputechconf.com/gtc/2012/presentations/S0514-GTC2012-GPU-Performance-Analysis.pdf)). In this sense, they’re related. 🙂

> It performs the same regardless of syntax `ldg(in,I)` , or `in[I]`
> 
> > It makes no difference because the global memory accessing pattern in your kernel is non-coalescing, and `in` is rarely hit in L1 cache.
> > 
> > > IIUC these reads are never cached in L1, but in L2 or in the read-only data cache depending on the use of `ldg` . I’m also not sure what memory coalescing has to do with that.

Using read-only loads or not should not perform exactly the same, but it looks like the performance difference is ignorable as OP reported above. I think the reason is just like the scenarios described on [S0514-GTC2012-GPU-Performance-Analysis page 44-45](http://on-demand.gputechconf.com/gtc/2012/presentations/S0514-GTC2012-GPU-Performance-Analysis.pdf).

In the case of caching load(`in[I] `), the bus utilization is 128/(N \* 128); in the case of read-only load(`ldg(in,I)`), the bus utilization is 128/(N \* 32). When N is quite large, the utilization difference can be ignored. The lower bus utilization means there are more “wasted” transactions which in turn affects the effective memory throughput.

That’s why I thought the main reason for the bad performance is due to the non-coalescing global memory accessing pattern rather than whether to use read-only cache or not.

---

<div class="post-metadata">

**Author:** ![Michael\_Benton](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@Michael\_Benton](https://discourse.julialang.org/u/Michael_Benton)\
**Post date:** [February 6, 2020, 11:56pm UTC](https://discourse.julialang.org/t/improving-cache-behavior-with-ldg/33628/7 "2020-02-06T23:56:02Z")

</div>

For someone else reading this, a good reference for how ldg works:

> **[Programming Guide :: CUDA Toolkit Documentation](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html)**
>
> The programming guide to the CUDA model and interface.