Flux relu fails with CUDA gpu

I’m getting error “CuArray only supports bits types” when using relu in code like:

model = Chain(Dense(2, 2, relu), Dense(2, 1, identity)) |> gpu

All works fine though with:

model = Chain(Dense(2, 2, (x) -> max(0, x), Dense(2, 1, identity)) |> gpu

Can anyone help how to fix relu or I should use below code?

Can you post the type and size of the input you’re passing to the model? ReLU should work just fine with as long as the inputs are floats:

julia> x = cu(rand(2, 4))
2×4 CuArray{Float32,2}:
 0.648422  0.723313  0.857359  0.612986
 0.575391  0.343899  0.735907  0.768486

julia> model = Chain(Dense(2, 2, relu), Dense(2, 1, identity)) |> gpu
Chain(Dense(2, 2, relu), Dense(2, 1))

julia> model(x)
1×4 CuArray{Float32,2}:
 -0.0808477  -0.0736467  -0.105517  -0.0689205

It is ‘Array{Tuple{CuArray{Float32,1},Float64},1}’

Note, it fails in train! method. Here is complete code (it does not work with relu, but if I replace it with ‘(x) -> max(o, x)’ it starts to work:

using Flux
using Flux: throttle
using CUDA

regX = rand(1000)
regY = 50 .+ 1000 * regX + 2 * randn(1000)
regZ = 10 .+ 5 .* regX + 12 .* regY .+ 0.2 * (randn(1000) .- 0.5) 

data = [([regX[i],regY[i]],regZ[i]) for i in 1:length(regX)]
data = gpu.(data)
println(typeof(data))

model = Chain(Dense(2, 2, relu), Dense(2, 1, identity)) |> gpu

loss(x, y) = Flux.mse(model(x), y) 
evalcb = () -> @show(sum([loss(i[1],i[2]) for i in data2D]))

opt = ADAM(0.01)
println(typeof(params(model)))

for i=1:10 
  Flux.train!(loss, params(model), data, opt, cb = throttle(evalcb, 1000)) 
end