Flux relu fails with CUDA gpu

New to Julia
1

I’m getting error “CuArray only supports bits types” when using relu in code like:

model = Chain(Dense(2, 2, relu), Dense(2, 1, identity)) |> gpu

All works fine though with:

model = Chain(Dense(2, 2, (x) -> max(0, x), Dense(2, 1, identity)) |> gpu

Can anyone help how to fix relu or I should use below code?

2

Can you post the type and size of the input you’re passing to the model? ReLU should work just fine with as long as the inputs are floats:

julia> x = cu(rand(2, 4))
2×4 CuArray{Float32,2}:
 0.648422  0.723313  0.857359  0.612986
 0.575391  0.343899  0.735907  0.768486

julia> model = Chain(Dense(2, 2, relu), Dense(2, 1, identity)) |> gpu
Chain(Dense(2, 2, relu), Dense(2, 1))

julia> model(x)
1×4 CuArray{Float32,2}:
 -0.0808477  -0.0736467  -0.105517  -0.0689205
3

It is ‘Array{Tuple{CuArray{Float32,1},Float64},1}’

Note, it fails in train! method. Here is complete code (it does not work with relu, but if I replace it with ‘(x) → max(o, x)’ it starts to work:

using Flux
using Flux: throttle
using CUDA

regX = rand(1000)
regY = 50 .+ 1000 * regX + 2 * randn(1000)
regZ = 10 .+ 5 .* regX + 12 .* regY .+ 0.2 * (randn(1000) .- 0.5) 

data = [([regX[i],regY[i]],regZ[i]) for i in 1:length(regX)]
data = gpu.(data)
println(typeof(data))

model = Chain(Dense(2, 2, relu), Dense(2, 1, identity)) |> gpu

loss(x, y) = Flux.mse(model(x), y) 
evalcb = () -> @show(sum([loss(i[1],i[2]) for i in data2D]))

opt = ADAM(0.01)
println(typeof(params(model)))

for i=1:10 
  Flux.train!(loss, params(model), data, opt, cb = throttle(evalcb, 1000)) 
end