# 2nd order optimization for Julia

**URL:** <https://discourse.julialang.org/t/2nd-order-optimization-for-julia/120966>\
**Category:** Machine Learning\
**Created:** [October 6, 2024, 6:21am UTC](https://discourse.julialang.org/t/2nd-order-optimization-for-julia/120966 "2024-10-06T06:21:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tarny\_GG\_Channie](https://avatars.discourse-cdn.com/v4/letter/t/3bc359/32.png) [@Tarny\_GG\_Channie](https://discourse.julialang.org/u/Tarny_GG_Channie)\
**Post date:** [October 6, 2024, 6:21am UTC](https://discourse.julialang.org/t/2nd-order-optimization-for-julia/120966/1 "2024-10-06T06:21:58Z")

</div>

Lately, I’ve seen second-order optimization suddenly becoming a trend in deep learning.  
Is this

1. An opportunity that Julia neural network system can do well?
2. Something that would rather shift the balance toward PyTorch/etc?
3. Just yet another development that doesn’t shift the landscape too much?

Julia is said to support high-order automatic differentiation. Now that 2nd order optimization is becoming more popular in neural network, is this an opportunity for Julia?

Here’s arguably the paper that started the trend.

> **[Shampoo: Preconditioned Stochastic Tensor Optimization](https://arxiv.org/abs/1802.09568)**
>
> Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning...

The landscape is changing fast.

---

<div class="post-metadata">

**Author:** ![gdalle](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gdalle/32/224680_2.png) [@gdalle](https://discourse.julialang.org/u/gdalle)\
**Post date:** [October 6, 2024, 10:34pm UTC](https://discourse.julialang.org/t/2nd-order-optimization-for-julia/120966/2 "2024-10-06T22:34:17Z")

</div>

Not really relevant to deep learning (because we’re still mostly CPU-focused) but the tryptic DifferentiationInterface.jl + SparseConnectivityTracer.jl + SparseMatrixColorings.jl allows you to [compute sparse Hessians effortlessly](https://gdalle.github.io/DifferentiationInterface.jl/DifferentiationInterface/stable/tutorials/advanced/#Sparsity), which is key for second-order methods on large-scale problems.