Speed Comparison Python v Julia for custom layers

[willleeney]

All ML libraries do this, because each layer depends on the output of the previous one.

This isn’t true. The example I gave is a model where this isn’t the case? It processes xi with one layer and xj with one layer, which isn’t sequential?

[ToucheSir]

Yes, that’s what I meant by “branching”. What I thought you meant was wanting to run a sequence of layers in parallel, which wouldn’t work because of the aforementioned data dependency.

@willleeney

Or I can implement my own layer in simplechains but it seems like to do this I would basically just be answering my question.

Maybe you could clarify what the question here is? If your custom layer uses SimpleChain layers much like your current implementation uses Flux layers, the library is still helping by providing fast layer building blocks, memory management utils, etc.

[willleeney]

wanting to run a sequence of layers in parallel, which wouldn’t work because of the aforementioned data dependency

I had thought that I do want to do this? Can I not run this (xi * m.Wq) at the same time as (xj * m.Wk) using multiple threads seeing as they don’t use any of the same resources?

@ToucheSir

Maybe you could clarify what the question here is? If your custom layer uses SimpleChain layers much like your current implementation uses Flux layers, the library is still helping by providing fast layer building blocks, memory management utils, etc.

So if I can just drop and drag in TurboDense{false}(identity, n_in) in place of Dense then that would be great. In the example GitHub - PumasAI/SimpleChains.jl: Simple chains, they wrap the layers in a chain and then init the parameters, and also calculate the gradients using valgrad!($g, $schain, $x, $p). Are you saying that I can simply replace the my dense layers within my custom block with this? I was under the impression that it’s not that simple?

[ToucheSir]

@willleeney

Can I not run this (xi * m.Wq) at the same time as (xj * m.Wk) using multiple threads seeing as they don’t use any of the same resources?

Yes you could, I was referring to something like having a 3-layer MLP (a sequence of layers) and wanting to run all 3 layers at the same time concurrently.

@willleeney

I was under the impression that it’s not that simple?

I haven’t written custom layers with SimpleChains myself so I’m not equipped to answer this question. Since it doesn’t show up in the docs, you might want to reach out to the maintainers with it.

[willleeney]

@willleeney

So if I can just drop and drag in TurboDense{false}(identity, n_in) in place of Dense then that would be great. In the example GitHub - PumasAI/SimpleChains.jl: Simple chains, they wrap the layers in a chain and then init the parameters, and also calculate the gradients using valgrad!($g, $schain, $x, $p). Are you saying that I can simply replace the my dense layers within my custom block with this? I was under the impression that it’s not that simple?

@Elrod Please may you shed some insight as the main contributor to SimpleChains.jl

[willleeney]

This issue is related as discusses/requests having the parallel layers which would help me create the custom layers like I want: Parallel Chains and Layers · Issue #84 · PumasAI/SimpleChains.jl · GitHub

This issue is related to a broader discussion on Flux compatibility which talks about having drop in turbo layers where a flux layer is expected. I’m not sure how to develop the custom layer using turbo layers still though. Compatibility with Flux · Issue #59 · PumasAI/SimpleChains.jl · GitHub

This issue is related to having two simple chains overwrite each other Two SimpleChains overwrite each other's predictions? · Issue #138 · PumasAI/SimpleChains.jl · GitHub So I’m guessing that having a SimpleChain of SimpleChains isn’t going to work.