I’m working on a new theory on spiking neural network using Julia, still work in progress, not released yet.
The problem of SNN is that it has not come to a consensus like normal NN, so you cannot have a standard framework for people to use…
As far as I know the people who use SNN has very different dataset/structure/training algorithm/computational model/purpose.
Some are trying to build neural simulator as realistic as possible, to mimic the pattern happen in brain.
Some are fitting a NN trained by traditional BP and transformed into SNN.
In term of computation, there are event based SNN, time-step based, rate based model, there are also people doing it by ODE solving. Most are just using CPU computation only. And most framework can only simulate very small number of neurons on a single computer.
In term of learning algorithm, some are experimenting with variations of STDP or Hebbs rule, on either spike train recorded from device or brain’s biological data. There are also people doing STDP learning on DL dataset like MNIST. And there are also people trying to get BP to work on SNN.
My personal goal on this project is to have a general learning theory on SNN learning, implemented with CUDA.jl(GPU acceleration), it’s a time-step based model, so it’s not as realistic as those event-based or ODE based model and has it’s limitation. But it’s fast and it allows me run experiment on larger scale of neurons, I believe with only neurons at a scale can show more interesting phenomenon.
But as a independent researcher, I’m doing it rather slow…
I don’t think we have anything close to NEURON, NEST or BRIAN in Julia. But I think the current Julia ecosystem would have a lot to offer to write a competitor, e.g. DifferentialEquations, Unitful or Automatic Differentiation (Zygote and friends).
That was exactly my thought, I am surprised that not much work has been done with AD and the ODE library.
I am interested in playing with recent learning algorithms like e-Prop and SG and different neuron models. Ideally something based on OrdinaryDiffEq, but I am not sure how big a network this could simulate.
For the simple (forward Euler) integration used by Bellec et al. in the e-prop paper you don’t necessarily need OrdinaryDiffEq, but I think you could use it (with Euler) without any performance penalty. The pseudo-derivatives of Bellec et al. are easy to implement with Zygote, e.g.
spike(u) = u .> 0
function Zygote.ChainRules.rrule(::typeof(spike), u)
output = spike(u)
function sp_pullback(y)
return NoTangent(), @.(y * .3 * max(0, 1 - abs(u)))
end
return output, sp_pullback
end
I think you just don’t need a SNN lib for what you want, in Julia, if you use CPU only, you’ll just need array of your structs and apply the algorithm on them. SNN doesn’t have too much line of code if ODE is well handled.
If you use CUDA, it’s still super easy to write kernels with CUDA.jl.
If you use FPGA or some spike-based hardware, I cannot help here…
I wrote WaspNet.jl, but it’s not something I use for work anymore. There is a partially implemented DifferentialEquations.jl branch for the integrator, but I never got it merged into master. I gave a small talk on the project at JuliaCon 2020 and Chris Rackauckas helped me get DiffEq up and going afterwards. With WaspNet.jl we were trying to support spiking, non-spiking, and some idea about “wrapped neurons” with functions applied to the spiking outputs, so DiffEq.jl was a bit harder to work with in the latter cases (or I was just using it wrong!).
I’ve been thinking about testing and merging the DiffEq branch to master, if there’s interest I could see about doing that in the near future.
I am interested in playing with different types of neurons, and it would be very natural to capture their behavior with ODEs. Not saying it has to be done this way and I don’t know at what point this would run into performance issues. Currently I am looking at all that is available.
In my framework, I define different type of neurons by it’s output effect(and all neurons is affected the same way by all kinds of neurons). So with julia’s multiple dispatch, I could implement it as a apply() method, dispatch on different type of neurons.
For my current work, I cares more about computational function than biological accuracy, so I choose to ignore some temporal feature of real neuron, and use time-step based model. The obvious benefit is that I can easily run networks with at least 10000 neurons on a single 2080ti GPU, and takes about 1-2ms for all neuron to run a single step. If you take 1 step as 1ms, I can almost get real time speed. So maybe in the future I can plug a live camera for it to see the world.(I didn’t really try to scale to the limit, so I don’t know how big it can hold)
Large scale computational function is more interesting than running accurate simulation on hundreds of neurons for me…
Im interested to see more about WaspNet’s integration to DifferentialEquations specially to compose them with NeuralPDE.jl for learning.
If I could help in any way let me know. I will be implementing STDP rules for learning in the meantime. If any of you have any advice or code you could share it would be amazing.
The DiffEq.jl integration is something that I started, but never finished implementing. I recently got my dev environment set up on my personal machine so I’ll see about cracking WaspNet.jl open again this weekend.
That being said, I don’t have much time to maintain that work anymore and no longer work in computational neuroscience. There is an open PR or two from someone that I need to get merged with master, but I can’t guarantee that DiffEq.jl integration will happen with any certainty. I seem to recall that branch working pretty well for spiking neurons, but non-spiking were more of an issue. If we relax WaspNet.jl to just spiking neurons it becomes more tractable, but it does remove a significant chunk of (questionably useful) functionality.
That would be amazing. I have the time and motivation to work in the integration of WaspNet into DiffEq if you experts lead the way. Let me know if you are interested.
Also a contributor is doing RL with waspnet so his input might be very valuable in order to have a good design without reimplementing the wheel.
I am using your SpikeSynchrony.jl package+SpikingNeuralNetworks.jl+Evolutionary.jl to optimize single cells and networks of spiking neural networks. It seems to work really well, because pairwise spike distance is a pretty good error signal if you want to fit models to spike times.
In the code block below, I compute the pairwise spike distance between a neuron in network A (sampled by an optimizer), and the corresponding neuron in network B (from a ground truth network).
@inbounds for (_, i) in zip(spkd, eachindex(spkd))
if !isempty(spkd0[i]) && !isempty(spkd_found[i])
maxt1 = findmax(spkd0[i])[1]
maxt2 = findmax(spkd_found[i])[1]
maxt = findmax([maxt1, maxt2])[1]
if maxt1 > 0.0 && maxt2 > 0.0
t, S = SpikeSynchrony.SPIKE_distance_profile(
unique(sort(spkd0[i])),
unique(sort(spkd_found[i]));
t0 = 0.0,
tf = maxt,
)
spkd[i] = SpikeSynchrony.trapezoid_integral(t, S) / (t[end] - t[1])
end
end
end
Do you have any ideas about the best way to parallelize the code? I tried one version, where spkd was a SharedArray, and @inbounds for (_, i) in zip(spkd, eachindex(spkd)) was written: @time @sync @distributed for (_, i) in zip(spkd, eachindex(spkd)). The code executed okay, but it was slower than the @inbounds version.
This is the result of a single cell optimization using the AdExp model:
Note this is a python plot. The blue Vector comes from an AdExp model that I wrote inside AStupid Bears SNN package.
Hi @sbuercklin, is the advantage of your DiffEq.jl branch of WaspNet.jl that it is somehow automatically compliant with CUDA.jl? If so, I’d be happy to try it out. I don’t care that the non-spiking neurons are broken. I am only interested in the spiking ones.
I wonder if anyone knows of a Computational Neuroscience lab that can create postdoc positions and favours Julia development?
@mefob in response to your initial post now there is also this package: https://github.com/darsnack/SpikingNN.jl, which has a branch: refactor/gpu-support. I have found GPU support (via CuArray) does not work on arbitrary models and networks, but it does work in a limited way on some models.
At the moment I am working on a repository that uses genetic algorithms to fit network raster plots and single-cell recordings for spike train data. The repository is here: https://github.com/russelljjarvis/SpikeNetOpt.jl. The code organization is currently a bit of a mess, as I am still learning how to make packages in Julia. Most often I optimize SNNs and single cell models implemented by AStupidBear/SpikingNeuralNetworks.jl, because at the time, this package was the most feature-rich (more ready to go plotting utils) and more easily hackable for a beginner.
These other two (previously mentioned) packages do seem to offer alternative and equally merited approaches and methods. Ideally, someone would be able to integrate the best of all three SNN packages:
AStupidBear/SpikingNeuralNetworks.jl (hackable, lots of functioning network examples, plotting works well possibly not written in CUDA compliant manner).
leaflabs/WaspNet.jl (putative GPU support?, not much plotting functions).
darsnack/SpikingNN.jl (has different and convenient methods for providing arbitrary external current inputs to spiking neural models, has some putative GPU support, as mentioned above).
My approach at the moment is to use Genetic Algorithms to make SNNs a better representation of experimental data, as this approach conforms to my background, but I would also like to re-apply the code for Learning 2 Learn (L2L) experiments.
GPU support would likely be easier with DiffEq.jl, but it’s not a guarantee. The way WaspNet.jl handles updates is to iterate over all of the neurons and calculate their updates serially; this won’t translate to a GPU directly, so we need to handle the calculation of updates separately.
There are two main reasons I see for moving to DiffEq.jl for the backend:
DiffEq.jl has a large set of integrators available, including Euler(). As it stands I rolled my own Euler integrator which, from what I remember, was how most SNN simulation was done. This worked, but it’s not fully general for something that I don’t see a reason not to solve in full generality
DiffEq.jl has a lot built into it beyond just solving a differential equation. Handling the solutions (membrane voltage traces + spikes) is a lot easier with this extra machinery, the interface is standardized, and it decouples the differential equation solving from the SNN specification.
All this being said, I don’t see any reason why GPU support wouldn’t be possible with a proper DiffEq.jl rewrite of WaspNet.jl, and that’s something that we also get easier with the DiffEq.jl backend. It requires solving how to calculate the update vector on the GPU, which I see as the real limiting factor given how we currently specify the problem.
This is very cool. It looks like NeuronBuilder is intended more for biologically plausible neurons like the Hodgkin-Huxley as opposed to abstract neurons like the LIF model. Does Conductor.jl works with these more abstract/approximate computational neuron models as well?
