# ANN: Announcing Unitary.jl, a differentiable parametrisation of the group of Unitary Matrices

**URL:** <https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917>\
**Category:** Machine Learning\
**Created:** [October 7, 2020, 8:01am UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917 "2020-10-07T08:01:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 7, 2020, 8:01am UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/1 "2020-10-07T08:01:44Z")

</div>

For our SumProductTransformation networks [https://arxiv.org/abs/2005.01297](https://arxiv.org/abs/2005.01297), we have created an invertible “Dense” transformations usual in neural networks. Our version features efficient inversion and efficient calculation of a determinant of a Jacobian (of course only where this operation makes sense, i.e. it is from a R^d → R^d). Our approach (detailed in the paper) relies on a representation and optimization of a dense in an SVD decomposed form, for which we needed differentiable parametrisation of the group of Unitary matrices. We have separated this functionality to a separate repo, which is now registered and you can freely use it with Flux / Zygote. [https://github.com/pevnak/Unitary.jl](https://github.com/pevnak/Unitary.jl).

For implementation of Dense layer, see [https://github.com/pevnak/SumProductTransform.jl/blob/master/src/layers/svddense.jl](https://github.com/pevnak/SumProductTransform.jl/blob/master/src/layers/svddense.jl), whichi roughly implements `Bijectors.jl` interface.

I would be happy if someone founds a use of this. We are currently working on a GPU version, but it will takes a bit of time. Meanwhile, reach me for possible enhancements or questions.

Tomas

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 7, 2020, 12:13pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/2 "2020-10-07T12:13:42Z")

</div>

This is great! I was just starting to do almost exactly Unitary.jl for a project right now, perhaps I won’t have to.

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [October 7, 2020, 1:00pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/3 "2020-10-07T13:00:11Z")

</div>

Cool package! I know absolutely nothing about the ML context, but just some remarks:

1. It’s pretty confusing to call these unitaries if you’re only doing real matrices. Why don’t you call them orthogonal?

2. You can get other parametrizations by using exponential mappings. They should probably be better in the sense that they deform the metric less.

3. If I understand correctly it appears you just want to optimize over orthogonal matrices. Optim.jl and Manopt.jl support this. But if I understand correctly your approach is only quadratic scaling, while the Riemannian optimization methods are usually cubic. I guess that’s the main point? CC the manifold people @kellertuer @mateuszbaran

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [October 7, 2020, 2:21pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/4 "2020-10-07T14:21:00Z")

</div>

> You can get other parametrizations by using exponential mappings.

I’m not sure it’s different from what they are doing. The Givens rotation is `exp(A)` where `A` is skew-symmetric with only a single nonzero upper-triangular entry, i.e., a basis for `SO(n)`.

---

<div class="post-metadata">

**Author:** ![antoine-levitt](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/antoine-levitt/32/4008_2.png) [@antoine-levitt](https://discourse.julialang.org/u/antoine-levitt)\
**Post date:** [October 7, 2020, 2:30pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/5 "2020-10-07T14:30:31Z")

</div>

Right, I was thinking in the Riemannian optimization setting where you take the exponential (or some other exponential-like mapping) of a full skew symmetric matrix (the gradient of the objective function projected onto the tangent plane)

---

<div class="post-metadata">

**Author:** ![mateuszbaran](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mateuszbaran/32/221842_2.png) [@mateuszbaran](https://discourse.julialang.org/u/mateuszbaran)\
**Post date:** [October 7, 2020, 5:46pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/6 "2020-10-07T17:46:51Z")

</div>

That’s interesting. I guess this approach should be faster than something similar to typical manifold gradient-based optimization? That is using the standard matrix representation of orthogonal matrices, projecting Euclidean gradient to Riemannian gradient and applying a retraction. Though I don’t know, maybe using QR retraction instead of exact `exp` would be faster and accurate enough here.

Would it make sense to try using matrix in QR decomposed form instead of SVD?

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 7, 2020, 6:09pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/7 "2020-10-07T18:09:06Z")

</div>

Actually, we have used QR as well, but was experimentally better in our application. I believe that it was partially caused by the fact that when angle in givens rotation is zero, it represents a diagonal matrix, which is nice in machine learning.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [October 7, 2020, 6:11pm UTC](https://discourse.julialang.org/t/ann-announcing-unitary-jl-a-differentiable-parametrisation-of-the-group-of-unitary-matrices/47917/8 "2020-10-07T18:11:36Z")

</div>

For a Unitary (or Orthogonal) matrix of dimension d, we use precisely d^(d-1)/2 parameters. The multiplication of matrix by this vector requires 4 times more multiplications and two times more additions (besides sin and cos).

I am sorry for confusing name. I guess I cannot change it, since the package is registered.