# Can I move an array asynchronously from main program to CUDA?
**URL:** https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976
**Category:** GPU
**Tags:** gpu, gpuarrays, cuda
**Created:** [September 27, 2024, 8:27pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976 "2024-09-27T20:27:11Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@joaquimg](https://discourse.julialang.org/u/joaquimg)
#### Post date: [September 27, 2024, 8:27pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/1 "2024-09-27T20:27:12Z")
I have an application that would benefit from passing an array asynchronously from the main program to my CUDA GPU while I am doing other computations on my CPU. The GPU would only be working on receiving the array.
Also, can threads help here?
---
### Author:  [@RomeoV](https://discourse.julialang.org/u/RomeoV)
#### Post date: [September 27, 2024, 11:49pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/2 "2024-09-27T23:49:47Z")
EDIT: Updated to reflect @maleadt’s comments.
I think we can do something like this.
```julia
using CUDA
# in this example, we want to multiply D*M batches on the GPU.
D = CuMatrix(rand(1_000, 1_000))
op = Base.Fix1(*, D)
M = rand(1_000, 10*1024)
CUDA.pin(M) # https://developer.nvidia.com/blog/how-optimize-data-transfers-cuda-cc/#pinned_host_memory
idx_batches = Iterators.partition(axes(M, 2), 1024)
# We make three channels: copying to GPU, doing our operation, and copying back.
# We use spawned Julia threads for each. Not sure if this is necessary.
ch_cpu_to_gpu = Channel{CuMatrix{Float64}}(; spawn=true) do ch
foreach(idx_batches) do idx
put!(ch, CuMatrix(M[:, idx]))
end
end
ch_op = Channel{CuMatrix{Float64}}(; spawn=true) do ch
foreach(ch_cpu_to_gpu) do rhs
put!(ch, op(rhs))
end
end
ch_gpu_to_cpu = Channel{Matrix{Float64}}(; spawn=true) do ch
foreach(ch_op) do res
put!(ch, Matrix(res))
end
end
```
We can now do something else on the CPU and get the gpu batches:
```julia
@time for batch in ch_gpu_to_cpu
sleep(0.1)
end
# > 1.013279 seconds (663 allocations: 125.015 MiB, 1.43% gc time)
```
Compared to just moving the memory, without actually computing the multiplication:
```julia
@time begin
for idx in idx_batches
Matrix(CuMatrix(arr[:, idx]))
sleep(0.1)
end
end
# > 1.089683 seconds (296 allocations: 156.258 MiB, 0.41% gc time)
```
There’s also Dagger.jl, but it didn’t work so well for me.
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [September 28, 2024, 11:07am UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/3 "2024-09-28T11:07:21Z")
This will probably not execute asynchronously because copies to and from pageable host memory (i.e., what Julia arrays are by default) are mostly synchronous. See [How to Overlap Data Transfers in CUDA C/C++ | NVIDIA Technical Blog](http://devblogs.nvidia.com/parallelforall/how-overlap-data-transfers-cuda-cc/), or the “Pinned memory” section of [Learning/Courses/AdvancedCUDA/part1/2-2-memory\_management.ipynb at main · JuliaGPU/Learning · GitHub](https://github.com/JuliaGPU/Learning/blob/main/Courses/AdvancedCUDA/part1/2-2-memory_management.ipynb). You want to make sure you use page-locked CPU memory, either by using CUDA’s HostMemory, or by `pin`ning the array.
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [September 28, 2024, 7:00pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/4 "2024-09-28T19:00:15Z")
> [@RomeoV](#):
>
> `import CUDA.Mem`
Note that the `CUDA.Mem` submodule has been deprecated in recent versions of CUDA.jl: [CUDA.jl 5.4: Memory management mayhem ⋅ JuliaGPU](https://juliagpu.org/post/2024-05-28-cuda_5.4/)
---
### Author:  [@RomeoV](https://discourse.julialang.org/u/RomeoV)
#### Post date: [September 28, 2024, 7:06pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/5 "2024-09-28T19:06:27Z")
Thanks @maleadt. I’ve updated my example according to your comments, and it looks like it works now. Still would be good to check with a nvidia profiler though probably.
---
### Author:  [@RomeoV](https://discourse.julialang.org/u/RomeoV)
#### Post date: [September 28, 2024, 7:18pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/6 "2024-09-28T19:18:02Z")
Actually it looks like we still get unpinned memory when we index into `M`?
```julia
M = rand(1_000, 1_000)
CUDA.pin(M)
CUDA.@profile CuMatrix(M)
"""
┌──────────┬────────────┬───────┬────────────────────────────────┐
│ Time (%) │ Total time │ Calls │ Name │
├──────────┼────────────┼───────┼────────────────────────────────┤
│ 99.35% │ 6.46 ms │ 1 │ [copy pinned to device memory] │
└──────────┴────────────┴───────┴────────────────────────────────┘
"""
```
see “copy pinned to device memory”, but
```julia
CUDA.@profile CuMatrix(M[:, 1:100])
"""
┌──────────┬────────────┬───────┬──────────────────────────────────┐
│ Time (%) │ Total time │ Calls │ Name │
├──────────┼────────────┼───────┼──────────────────────────────────┤
│ 21.83% │ 64.13 µs │ 1 │ [copy pageable to device memory] │
└──────────┴────────────┴───────┴──────────────────────────────────┘
"""
```
, see “copy pageable to device memory”, and similarly
```julia
CUDA.@profile CuMatrix(@view M[:, 1:100])
"""
┌──────────┬────────────┬───────┬──────────────────────────────────┐
│ Time (%) │ Total time │ Calls │ Name │
├──────────┼────────────┼───────┼──────────────────────────────────┤
│ 15.17% │ 64.61 µs │ 1 │ [copy pageable to device memory] │
└──────────┴────────────┴───────┴──────────────────────────────────┘
"""
```
So actually the memory pinning doesn’t help if we have to index afterwards, even with `@view`?
---
### Author:  [@maleadt](https://discourse.julialang.org/u/maleadt)
#### Post date: [September 30, 2024, 9:19am UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/7 "2024-09-30T09:19:52Z")
> [@RomeoV](#):
>
> `CuMatrix(M[:, 1:100])`
That’s a copying slice, not a view.
> [@RomeoV](#):
>
> `CuMatrix(@view M[:, 1:100])`
Yeah, that’s unfortunate. Right now, we only allow CuArray construction from Arrays, all other types (e.g. the SubArray here) are first copied to an Array, losing the pin. Could be a good addition to CUDA.jl, but IIRC we removed this at some point because of the huge number of ambiguities we ran into.
---
### Author:  [@RomeoV](https://discourse.julialang.org/u/RomeoV)
#### Post date: [December 15, 2024, 8:37pm UTC](https://discourse.julialang.org/t/can-i-move-an-array-asynchronously-from-main-program-to-cuda/119976/8 "2024-12-15T20:37:49Z")
I’ve found a way to copy slices of one big memory array, though use of `CartesianIndices`:
```julia
# host data, pinned to enable async transfer.
M_hst = rand(1_000, 1_000)
CUDA.pin(M_hst)
idx_hst = CartesianIndices((axes(M, 1), 201:300));
# device buffer
M_dev = CuArray{eltype(M)}(undef, size(M, 1), 100);
idx_dev = CartesianIndices((axes(M, 1), 1:100));
CUDA.@profile copyto!(M_dev, idx_dev, M_hst, idx_hst)
# yields
┌──────────┬────────────┬───────┬────────────────────────────────┐
│ Time (%) │ Total time │ Calls │ Name │
├──────────┼────────────┼───────┼────────────────────────────────┤
│ 61.96% │ 82.73 µs │ 1 │ [copy pinned to device memory] │
└──────────┴────────────┴───────┴────────────────────────────────┘
```
Note: We’re looking for this to read “copy _pinned_ to device memory” instead of “copy _pagable_ to device memory”.
Moving data back the same way also works:
```julia
# we also check the backwards transfer
M_dst = zeros(eltype(M), size(M))
CUDA.pin(M_dst)
CUDA.@profile copyto!(M_dst, idx_hst, M_dev, idx_dev)
# yields
┌──────────┬────────────┬───────┬────────────────────────────────┐
├──────────┼────────────┼───────┼────────────────────────────────┤
│ 53.73% │ 87.5 µs │ 1 │ [copy device to pinned memory] │
└──────────┴────────────┴───────┴────────────────────────────────┘
```