# \[pre-ANN\]: CuTensorOperations.jl

**URL:** <https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305>\
**Category:** Package Announcements\
**Created:** [August 8, 2019, 9:42am UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305 "2019-08-08T09:42:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![juthohaegeman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juthohaegeman/32/8620_2.png) [@juthohaegeman](https://discourse.julialang.org/u/juthohaegeman)\
**Post date:** [August 8, 2019, 9:42am UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/1 "2019-08-08T09:42:21Z")

</div>

[CuTensorOperations.jl](https://github.com/Jutho/CuTensorOperations.jl) provides support for tensor contractions and related operations with `CuArray` objects, i.e. it makes [TensorOperations.jl](https://github.com/Jutho/TensorOperations.jl) compatible with `CuArray` objects.

For now, it merely dispatches the necessary operations to a new experimental library of NVidia (thanks to the NVidia developers involved) [cuTENSOR](https://developer.nvidia.com/cuTensor) and does not provide a Julia/CUDAnative implementation. The basic functionality of cuTENSOR is wrapped in the custom/experimental branch [`ksh/tensor`](https://github.com/JuliaGPU/CuArrays.jl/pull/330) of [CuArrays.jl](https://github.com/JuliaGPU/CuArrays.jl) (thanks to @maleadt and @kslimes).

Given the current state, this is a pre-announcement, and CuTensorOperations.jl is not registered yet. It is more likely to become part of TensorOperations.jl in time.

Check out the README for more info, known (current) limitations and installation instructions. Experiment, and report any issues!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 8, 2019, 11:32am UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/2 "2019-08-08T11:32:01Z")

</div>

Nicely done! This is a great thing to see added to the ecosystem!

Does it need to be a different macro? It would be better if it was the same macro and just worked via dispatch. That way it could work in generic codes.

Also, do you plan on adding differentiation rule overloads for ChainRules.jl? It would be nice for these operations to be compatible with AD.

---

<div class="post-metadata">

**Author:** ![juthohaegeman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juthohaegeman/32/8620_2.png) [@juthohaegeman](https://discourse.julialang.org/u/juthohaegeman)\
**Post date:** [August 8, 2019, 12:37pm UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/3 "2019-08-08T12:37:27Z")

</div>

The same macro works for `CuArray`s. The new macro `@cutensor` will take `Array`s of the host, and transfer them to the GPU for you. Let me know if the explanation is not clear on this fact.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 8, 2019, 12:38pm UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/4 "2019-08-08T12:38:46Z")

</div>

> [@juthohaegeman](#):
>
> The same macro works for `CuArray` s. The new macro `@cutensor` will take `Array` s of the host, and transfer them to the GPU for you. Let me know if the explanation is not clear on this fact.

Nope, I just missed that. Awesome! Hope to see some good chain rules on this 🙂

---

<div class="post-metadata">

**Author:** ![juthohaegeman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juthohaegeman/32/8620_2.png) [@juthohaegeman](https://discourse.julialang.org/u/juthohaegeman)\
**Post date:** [August 8, 2019, 1:53pm UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/5 "2019-08-08T13:53:55Z")

</div>

> [@ChrisRackauckas](#):
>
> Nope, I just missed that. Awesome! Hope to see some good chain rules on this 🙂

So do I. There was some work on AD with TensorOperations.jl by other people, e.g. @mcabbott . Not sure if that is still alive, fully functional, integrated somewhere, compatible with `ChainRules.jl` ?

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [August 8, 2019, 3:29pm UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/6 "2019-08-08T15:29:19Z")

</div>

Nice to see, haven’t tried it yet but I will!

> [@juthohaegeman](#):
>
> work on AD with TensorOperations.jl

Yes, I had a very naiive approach up & running but have not revisited it. Probably easy to package up if there is interest. Was fiddling with a smarter way, but it doesn’t work yet.

I believe @under-Peter is working harder on a new package which includes AD, although parallel to TensorOperations.jl, not built on top.

---

<div class="post-metadata">

**Author:** ![under-Peter](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/under-peter/32/3626_2.png) [@under-Peter](https://discourse.julialang.org/u/under-Peter)\
**Post date:** [August 8, 2019, 3:57pm UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/7 "2019-08-08T15:57:51Z")

</div>

> [@mcabbott](#):
>
> I believe @under-Peter is working harder on a new package which includes AD, although parallel to TensorOperations.jl, not built on top.

Partially built on top since when possible I dispatch to `TensorOperations` but for the GPU we currently have a custom kernel.

As far as AD is concerned, my project works by naively switching arguments as described [here](https://giggleliu.github.io/2019/04/02/einsumbp.html) - I’m not sure if there are much better ways, although it’s possible that the optimal contraction order for the backwards-pass is not usually the reverse of the forward pass and one could probably smartly cache results. If anyone has resources for smarter AD for these operations, I’d be interested.

---

<div class="post-metadata">

**Author:** ![mcabbott](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mcabbott/32/6603_2.png) [@mcabbott](https://discourse.julialang.org/u/mcabbott)\
**Post date:** [August 10, 2019, 1:03pm UTC](https://discourse.julialang.org/t/pre-ann-cutensoroperations-jl/27305/8 "2019-08-10T13:03:08Z")

</div>

Sorry about getting the details wrong!

I have just made some quick packages for [old](https://github.com/mcabbott/TensorTrack.jl) and [new](https://github.com/mcabbott/TensorGrad.jl) gradient code. These are a bit rough, but perhaps they display some different kinds of naivety. OP pointed me to this paper [Format selector for 1310.8023](https://arxiv.org/format/1310.8023) which has matlab code for what sounds like a very similar problem, but I have not read it closely.