# \[ANN\] NeuronBuilder.jl: A differentiable neuronal simulator

**URL:** <https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743>\
**Category:** Package Announcements\
**Created:** [March 30, 2022, 1:54pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743 "2022-03-30T13:54:08Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndreaRH](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrearh/32/32809_2.png) [@AndreaRH](https://discourse.julialang.org/u/AndreaRH)\
**Post date:** [March 30, 2022, 1:54pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/1 "2022-03-30T13:54:08Z")

</div>

[_NeuronBuilder.jl_](https://github.com/Dhruva2/NeuronBuilder.jl) is a quick package for building small networks of detailed, conductance-based neurons out of ion channels and synapses. The idea is that it’s easy to use this template and add your own ion channels / synapses, with your choice of dynamics.

## What’s the point of this package?

- It’s very natural to create neurons from ion channels. You can do it here without specifying large systems of complicated differential equations, just by concatenating sets of ion channels with a soma.

- parameter estimation usually relies on zero-order optimisation algorithms which scale very poorly with the number of parameters

- having a neuronal simulation that’s differentiable makes it possible to do sensitivity analysis and construct cost functions (e.g. minimising certain features of interest, see the animation below ⬇) with which to do the parameter estimation.

- _NeuronBuilder_ uses ModelingToolkit to build a symbolic ODE system. This can then be solved with OrdinaryDiffEq or DifferentialEquations, taking advantage of the fast solvers they offer 👍

- it can also interface with other packages like ForwardDiff, making it suitable for differentiation! 🌟

- simulating single neurons with _NeuronBuilder_ takes no more than 0.1 seconds and differentiating in Julia is, as we all know, also fast 😉

## Example use case:

- Because the neuron is differentiable we can interface this with [_MinimallyDisruptiveCurves.jl_](https://github.com/SciML/MinimallyDisruptiveCurves.jl) to find changes in maximal conductances that maintain average intracellular calcium over the trajectory timespan.
- The parameters are the 7 maximal conductances of each ion channel.

![animation11](https://global.discourse-cdn.com/julialang/original/3X/b/2/b2543e6cb2344f0b233b7cbcf5b59e439c3ba1cd.gif)  
Code coming soon in here: [_MinimallyDisruptiveCurves.jl_](https://github.com/SciML/MinimallyDisruptiveCurves.jl)

* * *

## Note:

If you want a more flexible platform to build neuron models from basic components you should check out the more comprehensive package [Conductor.jl](https://github.com/wsphillips/Conductor.jl) and for an awesome C++/Matlab neuronal simulator see [Xolotl](https://github.com/sg-s/xolotl). Both of these were valuable resources for the development of our new package.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [March 30, 2022, 2:09pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/2 "2022-03-30T14:09:10Z")

</div>

How do you differentiate through the spikes? That seems really difficult.

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [March 30, 2022, 2:22pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/3 "2022-03-30T14:22:52Z")

</div>

Biological neuronal spikes are really just very sharp curves when simulated at small timesteps, so they are differentiable (although many simulators do not treat them as such); I assume this is what @AndreaRH is doing here?

---

<div class="post-metadata">

**Author:** ![jpsamaroo](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jpsamaroo/32/46804_2.png) [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)\
**Post date:** [March 30, 2022, 2:24pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/4 "2022-03-30T14:24:55Z")

</div>

Very cool package! I love seeing this work being done in Julia; the promise of efficiently simulating biologically-plausible neurons is what drew me away from MATLAB to Julia in the first place 😄

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [March 30, 2022, 4:15pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/5 "2022-03-30T16:15:54Z")

</div>

Amazing to see!

> [@AndreaRH](#):
>
> - parameter estimation usually relies on zero-order optimisation algorithms which scale very poorly with the number of parameters

Using differentiation with [https://diffeqflux.sciml.ai/dev/examples/prediction\_error\_method/](https://diffeqflux.sciml.ai/dev/examples/prediction_error_method/) would likely be helpful in this case.

---

<div class="post-metadata">

**Author:** ![AndreaRH](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrearh/32/32809_2.png) [@AndreaRH](https://discourse.julialang.org/u/AndreaRH)\
**Post date:** [April 1, 2022, 5:35pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/6 "2022-04-01T17:35:50Z")

</div>

Like @[**jpsamaroo**](https://discourse.julialang.org/u/jpsamaroo) said, technically, having a higher accuracy to your ode solution gives you a more accurate gradient. Here’s a relevant link about the numerics:

> [@When ForwardDiff-ing through an ODE is unstable](https://discourse.julialang.org/t/when-forwarddiff-ing-through-an-ode-is-unstable/49484/5):
>
> With the super accurate computation, it works! norm(grad) = 0.0009. slight_smile I should also note that it works with adjoint sensitivity analysis (with a normal Tsit5() solution) the numerical explosion in ForwardDiff doesn’t happen when I look at the L2 deviation of less ‘stiff’ species (i.e. Calcium, the second state, instead of voltage) I’m not sure what the lesson is to take from this. Maybe that Forward mode can potentially diverge if your error tolerances are high on a stiff system…

But for parameter estimation and MinimallyDisruptiveCurves accurate gradients might not be necessary, as long as you have a differentiable cost function that’s meaningful.

---

<div class="post-metadata">

**Author:** ![AndreaRH](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/andrearh/32/32809_2.png) [@AndreaRH](https://discourse.julialang.org/u/AndreaRH)\
**Post date:** [April 1, 2022, 5:40pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/7 "2022-04-01T17:40:46Z")

</div>

Thanks! 🙂

Yes, the longer term reason for having this package was to explore which types of cost functions differentiably express behaviours we are scientifically interested in, such as bursting, spiking with a particular frequency, etc.

The PEM method is indeed really relevant. We actually have a person in our group working on implementation of a specialised PEM method that takes advantage of the regularities in conductance based neural models (see [Feedback identification of conductance-based models - ScienceDirect](https://www.sciencedirect.com/science/article/pii/S0005109820304969)).

the loss functions of e.g. L2 error on neural models have the same features (multiple disingenuous local minima, very sharp global minimum) as on your pendulum example in the link you said, and using PEM seems to prevent that happening, for reasons that are more rigorously explained in the paper i linked.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 2, 2022, 10:21am UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/8 "2022-04-02T10:21:49Z")

</div>

> [@AndreaRH](#):
>
> But for parameter estimation and MinimallyDisruptiveCurves accurate gradients might not be necessary, as long as you have a differentiable cost function that’s meaningful.

They are. Without accurate gradients you tend to get stuck in worse local minima and such. More accurate gradients are very helpful in parameter estimation, and in fact I usually find that one needs to lower tolerances when doing estimation.

---

<div class="post-metadata">

**Author:** ![Dhruva2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dhruva2/32/18475_2.png) [@Dhruva2](https://discourse.julialang.org/u/Dhruva2)\
**Post date:** [April 4, 2022, 9:11am UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/9 "2022-04-04T09:11:25Z")

</div>

That’s interesting. It goes counter to my intuition. I would have thought some noise in the gradient means you bounce out of a bad local minimum, where a perfect gradient might get you stuck at it (ignoring second order methods for now). For neural networks, stochastic gradient descent works pretty well.

I haven’t (yet) tried much parameter estimation where the objective function involves the solution of a differential equation. I guess there must be something different about the shape of the ‘typical’ loss functions, and/or the statistics of the gradient errors from inaccurate gradients, that implies your comment. I don’t have an intuition as to what these could be though…any relevant resources or intuitions you could provide? Thanks 🙂

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [April 4, 2022, 4:09pm UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/10 "2022-04-04T16:09:26Z")

</div>

One bad gradient can be pretty disruptive. It can shoot you right out of a parameter space by accidentally being really large. This happens because when the tolerance is low the space becomes bumpy, and bumps have a very large gradient. So even with local minima, accurate gradients are necessary to ensure you do completely fly out, and then things like ADAM can help something with “accurate enough” gradients converge well. I don’t know of any good papers on this, but we have lots of examples (especially in clinpharm from Pumas) demonstrating this phenomena.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [April 5, 2022, 6:57am UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/11 "2022-04-05T06:57:57Z")

</div>

Gradient clipping ([Gradient Clipping Explained | Papers With Code](https://paperswithcode.com/method/gradient-clipping)) is a popular technique to avoid being flung far away when encountering a large gradient.

---

<div class="post-metadata">

**Author:** ![Dhruva2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dhruva2/32/18475_2.png) [@Dhruva2](https://discourse.julialang.org/u/Dhruva2)\
**Post date:** [April 5, 2022, 9:18am UTC](https://discourse.julialang.org/t/ann-neuronbuilder-jl-a-differentiable-neuronal-simulator/78743/12 "2022-04-05T09:18:25Z")

</div>

thanks for the intuition both