Non-convex loss functions with Flux.jl

[mcabbott]

@MattSainsbury-Dale

However, even with P = 0.9 P=0.9P = 0.9 , which is only slightly different to the well-behaved absolute-error loss, some parameters of the neural network become NaN

Note that the gradient is quite different near to zero, x^-0.1 diverges. That could be what goes wrong?

You could try something like (abs.(ŷ .- y) .+ 1f-6).^P to move slightly away from the point. You could also try P = 1.1 to check