# Scalar Indexing with CUDA
**URL:** https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379
**Category:** General Usage
**Tags:** cuda
**Created:** [June 9, 2024, 6:43am UTC](https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379 "2024-06-09T06:43:37Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Jason\_Meziere](https://discourse.julialang.org/u/Jason_Meziere)
#### Post date: [June 9, 2024, 6:43am UTC](https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379/1 "2024-06-09T06:43:37Z")
I have a portion of my code that I haven’t quite figured out how to do without scalar indexing. I was messing around in the repl and noticed that, while something like
```julia
a[1]=1
```
will give the scalar indexing warning, something like
```julia
a[1:1].=1
```
does not. My question is:
Will these both have the same performance hit that comes from using scalar indexing, or is there something different about the second compared to the first in terms of performance?
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [June 9, 2024, 11:26am UTC](https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379/2 "2024-06-09T11:26:34Z")
The performance of both will be bad in that there’s quite some overhead associated with fetching GPU memory, much larger than the actual transfer time of a single element. The reason that we don’t error on the second expression, is that we simply detect scalar transfers by hooking the relevant scalar `getindex` methods, and not the vectorized one you’re using in the second example.
If you know that the performance overhead of this operation isn’t problematic (e.g., because you only perform it rarely), you can annotate the expression with `CUDA.@allowscalar`. If you _do_ need frequent scalar accesses to GPU memory, e.g. because you’re porting a CPU application but haven’t ported all the algorithms yet, consider using unified memory (see [CUDA.jl 5.4: Memory management mayhem ⋅ JuliaGPU](https://juliagpu.org/post/2024-05-28-cuda_5.4/#unified_memory_iteration)).
---
### Author:  [@Jason\_Meziere](https://discourse.julialang.org/u/Jason_Meziere)
#### Post date: [June 9, 2024, 2:06pm UTC](https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379/3 "2024-06-09T14:06:09Z")
I see, that makes sense. This is more for my personal understanding now, but I do have a few more questions.
What is the difference between a `getindex` on the cpu and one that is compiled into a kernel? Is it that in a kernel, the `getindex` is compiled and run on the gpu as well?
Also, how many elements would it take for the `setindex` command to be worth it in a simple assignment command (i.e. `a[1:n] .= 1`)?
Finally, does this change in the case of a view, such as `@views b .= a[1:n]`?
---
### Author:  [@de-souza](https://discourse.julialang.org/u/de-souza)
#### Post date: [June 9, 2024, 2:57pm UTC](https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379/4 "2024-06-09T14:57:33Z")
In NNlib.jl for example, the [gather method](https://fluxml.ai/Flux.jl/v0.14.15/models/nnlib/#NNlib.gather) seems to work by compiling a call to `getindex` into a GPU kernel:
>
Would it be faster to use unified memory instead of calling this method?
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [June 10, 2024, 8:04am UTC](https://discourse.julialang.org/t/scalar-indexing-with-cuda/115379/5 "2024-06-10T08:04:01Z")
> [@Jason\_Meziere](#):
>
> What is the difference between a `getindex` on the cpu and one that is compiled into a kernel? Is it that in a kernel, the `getindex` is compiled and run on the gpu as well?
Correct; `getindex` in a kernel is fast, it’s only slow when executed from the CPU where each access needs to fetch memory from the device. If it’s already executing on the GPU (i.e. in a kernel), these accesses are fast.
> [@de-souza](#):
>
> In [NNlib.jl](https://juliahub.com/ui/Packages/General/NNlib) for example, the [gather method](https://fluxml.ai/Flux.jl/v0.14.15/models/nnlib/#NNlib.gather) seems to work by compiling a call to `getindex` into a GPU kernel
Not necessarily. Unified memory comes with its own drawbacks too. For one, you’re now relying on the kernel driver to do the memory management, which may be suboptimal. The operation also becomes synchronous, i.e., blocking while the page fault is being handled, while a regular kernel launch is asynchronous and makes it possible for the CPU to do other work while waiting for it to complete.