# Help using CUDA, Zygote, and random numbers

**URL:** <https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458>\
**Category:** Machine Learning\
**Tags:** cuda, zygote\
**Created:** [December 4, 2024, 2:28pm UTC](https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458 "2024-12-04T14:28:47Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![bgctw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgctw/32/22050_2.png) [@bgctw](https://discourse.julialang.org/u/bgctw)\
**Post date:** [December 4, 2024, 2:28pm UTC](https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458/1 "2024-12-04T14:28:47Z")

</div>

I get the error “`llvmcall` requires the compiler” when trying to take the gradient of a function that involves generating random numbers in CUDA. Here is a minimal example:

```julia
using GPUArraysCore: GPUArraysCore
using CUDA, Flux
using LinearAlgebra
using Zygote

function f3(v::AbstractVector{T}) where {T}
    randn(T, 4,4) * v[1:4]
end
function f3(v::GPUArraysCore.AbstractGPUVector{T}) where {T}
    CUDA.randn(T, 4,4) * v[1:4]
end
v_orig = collect(1.0:10.0)
Zygote.gradient(v -> sum(f3(v)), v_orig) # works

v = v_orig |> gpu
m = f3(v)
Zygote.gradient(v -> sum(f3(v)), v) # fails
Zygote.gradient(v -> sum(cpu(f3(v))), v) # fails

```

I suspect, I did not sufficiently understand the CUDA/Zygote workings yet. Could someone, please, explain to me why this fails, what I need to do, and point me to the resources to understand better?  
Background for generating random numbers: I want to use a Monte-Carlo approximation of an expectation inside a cost function of a stochastic gradient descent.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/27854_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [December 4, 2024, 2:50pm UTC](https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458/2 "2024-12-04T14:50:34Z")

</div>

Not sure why this fails but can you maybe generate the random numbers outside of your differentiated function and pass them as arguments?

---

<div class="post-metadata">

**Author:** ![bgctw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgctw/32/22050_2.png) [@bgctw](https://discourse.julialang.org/u/bgctw)\
**Post date:** [December 4, 2024, 3:41pm UTC](https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458/3 "2024-12-04T15:41:49Z")

</div>

Yes, thanks, passing in another pre-allocated CuArray of randon numbers works, that I have tried before.

However, it is awkward and probably not very efficient to pre-allocate a lot of random-data and pass it with a `DataLoader` of a machine-learning optimization to the cost function. There are much fewer observations, covariates, and parameters compared the number of randoms that I need. Maybe, I need to implement a special `DataLoader` that generates the random-numbers when asked for the next batch.

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [December 4, 2024, 5:17pm UTC](https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458/4 "2024-12-04T17:17:47Z")

</div>

What’s probably happening is that Zygote is trying to differentiate the code inside `CUDA.randn`, which ultimately calls non-Julia code via `llvmcall`.

The reason that it does not try to do this with `randn` is that there’s a rule instructing it not to look, [here](https://github.com/JuliaDiff/ChainRules.jl/blob/e05500931b5424897d49ae184ed8e796e6e5357f/src/rulesets/Random/random.jl#L32).

You can define such a rule for `CUDA.randn` in your code, or make a PR adding it for everyone [here](https://github.com/JuliaGPU/CUDA.jl/blob/master/ext/ChainRulesCoreExt.jl).

In general, you can also tell Zygote to ignore some bit of code by doing this (or the ChainRulesCore equivalent):

```julia
r = Zygote.@ignore CUDA.randn(T, 4,4)
r * v[1:4]

```

---

<div class="post-metadata">

**Author:** ![bgctw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgctw/32/22050_2.png) [@bgctw](https://discourse.julialang.org/u/bgctw)\
**Post date:** [December 23, 2024, 8:59am UTC](https://discourse.julialang.org/t/help-using-cuda-zygote-and-random-numbers/123458/5 "2024-12-23T08:59:15Z")

</div>

Thanks for this guide.

With the undocumented and deprecated-warning  
`Zygote.@ignore` ignore approach, I can continue developing. And a more lasting general solution at the `ChainRulesCore` repo is in progress.