Speed Comparison Python v Julia for custom layers

Specific Domains Machine Learning
, , , ,
13

Sorry, my bad, no. It should do it for reverse. But I am anyway surprised that the compiler does not compile away redundant operations, because this is something what it should do. May-be, defining a special forward pass for Dense without activation would do the job.

function (a::Dense{<:Any, <:Any, ::Bool})(x::AbstractVecOrMat)
  _size_check(a, x, 1 => size(a.weight, 2))
  σ = NNlib.fast_act(a.σ, x)  # replaces tanh => tanh_fast, etc
  xT = _match_eltype(a, x)  # fixes Float64 input, etc.
  return σ.(a.weight * xT)
end

If the difference is so substantial, it might be interesting to add it.

1 Like