@aramirezreyes, have you looked at FastConv.jl?
The package seems to be broken but looking at the code and description they seem to implement the same idea of performing direct convolution with small kernels as in the code posted by @sijo here.
From your posts above it seems that you had tried something similar but if you could please confirm, it would be helpful.