# FiniteDiff.jl: Fast (Sparse) Gradients, Jacobians, Hessians!

**URL:** <https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976>\
**Category:** Package Announcements\
**Tags:** differentiation\
**Created:** [January 4, 2020, 6:59pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976 "2020-01-04T18:59:33Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 4, 2020, 6:59pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/1 "2020-01-04T18:59:34Z")

</div>

FiniteDiff.jl is a new Julia library for fast gradients, Jacobians, and Hessians which supports sparsity and GPUs!

> **[GitHub - JuliaDiff/FiniteDiff.jl: Fast non-allocating calculations of gradients,...](https://github.com/JuliaDiff/FiniteDiff.jl)**
>
> Fast non-allocating calculations of gradients, Jacobians, and Hessians with sparsity support

Okay, I lied, it’s not an entirely new library. It’s the next incarnation of what was known as DiffEqDiffTools, but the library essentially had nothing DiffEq left in it so it was moved over to JuliaDiff and turned into FiniteDiff.jl. While Calculus.jl also can give you Jacobians and Hessians, FiniteDiff.jl allows you to use non-allocating mutating forms by pre-allocating caches, and is compatible with StaticArrays. You can use SparseDiffTools (now moved to JuliaDiff) to get color vectors and specialize the calculations on sparsity patterns.

But the most special part of FiniteDiff.jl is that it can utilize special matrix types to build fast differencing algorithms on them. For example, if we have a BlockBandedMatrix from [BlockBandedMatrices.jl](https://github.com/JuliaMatrices/BlockBandedMatrices.jl) we can have the Jacobian calculation optimized on this exact type in terms of differentiation directions and iteration schemes. For example, from a system of PDEs we can do:

```julia-auto
function pde(out, x)
	x = reshape(x, 100, 100)
	out = reshape(out, 100, 100)
	for i in 1:100
		for j in 1:100
			out[i, j] = x[i, j] + x[max(i -1, 1), j] + x[min(i+1, size(x, 1)), j] + x[i, max(j-1, 1)] + x[i, min(j+1, size(x, 2))]
		end
	end
	return vec(out)
end
x = rand(10000)
using FillArrays, BlockBandedMatrices
Jbbb = BandedBlockBandedMatrix(Ones(10000, 10000), fill(100, 100), fill(100, 100), (1, 1), (1, 1))
colorsbbb = ArrayInterface.matrix_colors(Jbbb)
bbbcache = FiniteDiff.JacobianCache(x,colorvec=colorsbbb,sparsity=Jbbb)
FiniteDiff.finite_difference_jacobian!(Jbbb, pde, x, bbbcache)

```

While a naive Jacobian calculation like that in FiniteDifferences.jl or Calculus.jl would:

- Allocate
- Iterate the matrix as a dense matrix (ouch!)
- Use 10000 calls to `pde`

This one only allocates the `cache` once and is non-allocating in all proceeding Jacobian calculations, specializes its iteration scheme on the matrix, and **only uses 10 `pde` calls to fill the entire Jacobian**. Did I hear a “zoom zoom”? 🚘 🚀. I don’t think I know of packages in other languages that not only support sparse matrices, but structured matrices as well.

Special thanks to @YingboMa @oxinabox @dlfivefifty Langwen Pankaj @dextorious who have all been fundamental in this project, as well as @pkofod who was an early adopter in NLsolve.jl/Optim.jl. Special thanks to @jlperla who has helped make sure this project reached these goals.

Our next steps is to add sparse Hessian support. If you find this project useful, please star it. Let’s move all of the main numerical Julia packages away from the older allocating Calculus.jl and start zooming 🙂.

## Downstream Support

Quick note on library authors who want to start adding downstream sparsity support. All you really need from the users is `colorvec` and `sparsity` arguments. `sparsity` is the sparsity pattern of the user’s function (or it can be automatically generated using [SparsityDetection.jl](https://github.com/JuliaDiffEq/SparsityDetection.jl)) and `colorvec` is the color vector, usually generated by [SparseDiffTools.jl](https://github.com/JuliaDiff/SparseDiffTools.jl). DifferentialEquations.jl’s interface on sparsity is thus, “give me your `sparsity` pattern and your `colorvec` and I’ll specialize the internal Jacobian calculations on this!”. NLsolve.jl, Optim.jl, etc. can all support this just by adding these two arguments and forwarding them to these functions. I hope all of Julia makes use of this and we supported sparsity everywhere soon. 2020: the year of Julialang sparsity.

Also note the SpraseDiffTools has a version of forward-mode AD which also uses those same two arguments, so it’s an extension of ForwardDiff that gives you sparse ForwardDiff with `sparsity` and `colorvec`.

---

<div class="post-metadata">

**Author:** ![daniel](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/daniel/32/6480_2.png) [@daniel](https://discourse.julialang.org/u/daniel)\
**Post date:** [January 4, 2020, 7:16pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/2 "2020-01-04T19:16:32Z")

</div>

This is super cool! I’m not the target group for this, but the fact that this is even possible justified an instant star. Zoom zoom

---

<div class="post-metadata">

**Author:** ![cscherrer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cscherrer/32/7631_2.png) [@cscherrer](https://discourse.julialang.org/u/cscherrer)\
**Post date:** [January 6, 2020, 4:42am UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/3 "2020-01-06T04:42:56Z")

</div>

This is great, looking forward to trying out sparse structured Hessians for hierarchical Bayesian models!

---

<div class="post-metadata">

**Author:** ![pkofod](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pkofod/32/2179_2.png) [@pkofod](https://discourse.julialang.org/u/pkofod)\
**Post date:** [January 6, 2020, 7:02am UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/4 "2020-01-06T07:02:28Z")

</div>

We’ll be switching over this week. Thanks.

---

<div class="post-metadata">

**Author:** ![DrPapa](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/drpapa/32/6835_2.png) [@DrPapa](https://discourse.julialang.org/u/DrPapa)\
**Post date:** [January 6, 2020, 12:46pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/5 "2020-01-06T12:46:17Z")

</div>

Awesome stuff as always @ChrisRackauckas. Congrats to you and your team.

---

<div class="post-metadata">

**Author:** ![qwerytyui1234](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/qwerytyui1234/32/31952_2.png) [@qwerytyui1234](https://discourse.julialang.org/u/qwerytyui1234)\
**Post date:** [January 6, 2020, 4:51pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/6 "2020-01-06T16:51:36Z")

</div>

Hey Chris,

Thanks for this package. I wanted to try it out, but I get the following error message. Let me know whether I should open an issue on Github instead of cluttering up the thread here.

Would you happen to have any ideas? I am using Julia 1.1.0 and Juno 0.7.2. Is my Julia version the issue here? Thanks in advance.

```julia
ERROR: Unsatisfiable requirements detected for package FiniteDiff [6a86dc24]:
 FiniteDiff [6a86dc24] log:
 ├─possible versions are: 2.0.0 or uninstalled
 ├─restricted to versions * by an explicit requirement, leaving only versions 2.0.0
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left

```

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 6, 2020, 8:34pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/7 "2020-01-06T20:34:09Z")

</div>

It requires Julia 1.2 because of BlockBandedMatrices.jl stuff.

---

<div class="post-metadata">

**Author:** ![carstenbauer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/carstenbauer/32/4981_2.png) [@carstenbauer](https://discourse.julialang.org/u/carstenbauer)\
**Post date:** [January 7, 2020, 7:09am UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/8 "2020-01-07T07:09:00Z")

</div>

Just to be sure, in your example it seems like `bbbcache` is never used. Could this perhaps be a typo in the last line?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [January 7, 2020, 1:32pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/9 "2020-01-07T13:32:40Z")

</div>

Yeah typo fixed.

---

<div class="post-metadata">

**Author:** ![andrewning](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrewning/32/3661_2.png) [@andrewning](https://discourse.julialang.org/u/andrewning)\
**Post date:** [January 10, 2020, 5:22pm UTC](https://discourse.julialang.org/t/finitediff-jl-fast-sparse-gradients-jacobians-hessians/32976/10 "2020-01-10T17:22:18Z")

</div>

This is great! Just last month I was wishing for a package that did exactly this (coloring with AD). Our use is optimization and SparseDiffTools.jl sped up some of our Jacobians significantly. Thanks!