Is there a easy explanation or “rule of thumb” about which computations are run on the GPU?
For example, if there is a matrix ‘W’ and array ‘a’, both on the gpu, I am sure that W*a is computed on the GPU. But what about other statements such as
a = 2.f0 * a a[1:10] = 0 for i=1:10 a[i] = sin(i * 0.1f0)
Do some of these require bring the array back to the CPU, computing, and sending back?
This would be slower than just computing on the CPU.
Does the computation ever silentlly get sent to the CPU, compute, and send back?
The question is motivated by Flux (used CuArray) and Knet, however interested to understand more broadly.