That’s nice. Fused kernels are recommended for performance critical operations like the ReLU and Sigmoid activation functions and their gradients. It seems like the kernels are where I should draw my attention to first.
I was wrong about kernels, they’re not relevant for a single machine for now. Mathematical operations is my first step. The first file I’m working on is math_ops.jl, where I am defining type TensorArray for function add(x, y, name=None) which should return a tensor from tensors x + y elementwise, without broadcasting for now. Eventually I’ll have a question and post it here. Some action. @kristoffer.carlsson, I understand you’re well versed in tensors, any advice is welcome.
Great with some code. How do I run it? It seems that https://github.com/hpoit/LSTM/blob/master/TensorArray_ops.jl is filled with syntax errors (like no end for type, or no end for function). You are also using types that you haven’t defined like Tensor and Scalar.
Thanks! I just added type Scalar and type Tensor, but no ends yet. I’ll only come back here after I can give a textbook definition of the types and functions. I looked at your code, it’s pretty clean, and I’m happy about that. One question that emerged is in e.g.
function readt(index::Tensor{Int32}, name=None)
should it be name=None or name::String and None be specified inside the function? That’s probably in the Julia Doc but I haven’t reached it yet. I.e. should default values be defined in the argument or in the function? I pulled None out of Python and the Google Doc but to me it doesn’t make sense to specify a default value in the argument line and leave out ::String, unless there’s a way to specify both together as an argument?
Yes, a keyword arg with type and default value, name::String=None 
I’m not sure None is the most proper default value though. I’ll find out in a bit.