# Module idea for helping calculate large sparse Jacobians

**URL:** <https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941>\
**Category:** Optimization (Mathematical)\
**Created:** [August 24, 2019, 4:13pm UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941 "2019-08-24T16:13:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![xor0110](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xor0110/32/7926_2.png) [@xor0110](https://discourse.julialang.org/u/xor0110)\
**Post date:** [August 24, 2019, 4:13pm UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941/1 "2019-08-24T16:13:54Z")

</div>

I’ve put together a minimal, tiny draft of a project idea I had. It’s about helping to work with large structured models, the kind of thing you might use g2o for (and I imagine similar things might exist for FEM). Basically we have formulas that need to be evaluated for many indices, and the idea is to help doing that and also calculating the Jacobian.

I would love to hear some comments! Check it out at: [https://github.com/nlw0/SparseBlockJacobians.jl](https://github.com/nlw0/SparseBlockJacobians.jl)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 24, 2019, 11:04pm UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941/2 "2019-08-24T23:04:38Z")

</div>

Just FYI, in Julia `master` [we now have `AbstractSparseMatrixCSC` interface](https://github.com/JuliaLang/julia/pull/33039) (as an example usage, see also [https://github.com/tkf/SparseXX.jl/pull/2](https://github.com/tkf/SparseXX.jl/pull/2)). If you want to bundle block information in a custom sparse matrix type, it could be useful. (Although simple function-based API is the best approach sometimes.)

Note that this interface is still not “official.” The details would be finalized in:

[https://github.com/JuliaLang/julia/pull/33054](https://github.com/JuliaLang/julia/pull/33054)

---

<div class="post-metadata">

**Author:** ![xor0110](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xor0110/32/7926_2.png) [@xor0110](https://discourse.julialang.org/u/xor0110)\
**Post date:** [August 25, 2019, 11:23am UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941/3 "2019-08-25T11:23:37Z")

</div>

Thanks, I definitely have to learn more about what is going on regarding sparse arrays. I’m not so sure how this connects to what I’m looking for, though. Right now I’m just filling up a `SparseMatrixCSC` and at most would like a CSR version of it. The essence of the project is that we have models defined by a few formulas that involve few variables and they are applied over long vectors to define the full model. The main questions in my head right now are: what’s a good, declarative way to represent the model and its functions, how to ensure an efficient Jacobian update, and how to interface nicely with AD tools. Perhaps my liberal use of the term “block” is misleading about the emphasis?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [August 25, 2019, 11:46am UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941/4 "2019-08-25T11:46:41Z")

</div>

I hope there’s a bunch of iteration and coloring ideas in that interface. See [https://github.com/JuliaDiffEq/ArrayInterface.jl](https://github.com/JuliaDiffEq/ArrayInterface.jl)

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 25, 2019, 7:34pm UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941/5 "2019-08-25T19:34:04Z")

</div>

I saw you are passing block sizes to `update_jacobian_values!` and `update_jacobian_indices!`. I just thought it may make sense to embed this information in the matrix so that you only need to pass functions. Of course, as I said, a function-based API that manipulates `SparseMatrixCSC` directly totally makes sense as well.

---

<div class="post-metadata">

**Author:** ![tkf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tkf/32/17635_2.png) [@tkf](https://discourse.julialang.org/u/tkf)\
**Post date:** [August 25, 2019, 7:43pm UTC](https://discourse.julialang.org/t/module-idea-for-helping-calculate-large-sparse-jacobians/27941/6 "2019-08-25T19:43:57Z")

</div>

It’s just a dead simple CSC matrix. I think the only thing that is relevant to ArrayInterface is that you can write something like `has_sparsestruct(x::Type{<:AbstractSparseMatrixCSC})` instead of `has_sparsestruct(x::Type{<:SparseMatrixCSC})`.