# Stochastic differential equation with high dimensions is using too much memory and taking too long

**URL:** <https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478>\
**Category:** Numerics\
**Tags:** differentialequation\
**Created:** [July 11, 2023, 10:14am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478 "2023-07-11T10:14:35Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 11, 2023, 10:14am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/1 "2023-07-11T10:14:35Z")

</div>

Hey! I want to integrate a system of 40+ differential equations each with independent additive white noise. With the current implementation I’m trying this is using way too much memory and is too slow. A minimal example of this is:

```julia
using StochasticDiffEq, DiffEqNoiseProcess, OrdinaryDiffEq

function system!(du, u, p, t)
    du .= 0
end

function noise!(du, u, p, t)
    noise = p[1]
    du .= noise
end

function test_memory_usage()
    N = 40
    noise = 0.1
    ps = [noise]
    u0s = ones(Float64, N)
    tend = 50000.0
    ttrans = 0.0 
    noise_seed = 1
    alg = EM()
    dt = 0.01
    Δt = 10.0
    
    #1. For reference, integrate without noise 
    prob_ode = ODEProblem(system!, u0s, (0.0, tend), ps)
    sol_ode = @time solve(prob_ode, Euler(); dt, saveat=ttrans:Δt:tend)

    #2. with the Wiener Process it's way too slow 
    W = WienerProcess(0.0, zeros(length(u0s)))
    prob_independent_noise = SDEProblem(system!, noise!, u0s, (0.0, tend), ps, noise=W)
    sol_inde = @time solve(prob_independent_noise, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed);  

    #3. in place doesn't help
    W = WienerProcess!(0.0, zeros(length(u0s)))
    prob_independent_noise_inplace = SDEProblem(system!, noise!, u0s, (0.0, tend), ps, noise=W)
    sol_inde_in = @time solve(prob_independent_noise_inplace, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed);  

    #4. without the Wiener Processes it's way faster, but not the same result.
    prob_independent_noise_2 = SDEProblem(system!, noise!, u0s, (0.0, tend), ps)
    sol_inde_2 = @time solve(prob_independent_noise_2, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed);   

    #5. Single process, with dependent noise is also fast. 
    W = WienerProcess(0.0, 0.0)
    prob_dependent_noise = SDEProblem(system!, noise!, u0s, (0.0, tend), ps, noise=W)
    sol_de = @time solve(prob_dependent_noise, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed) 

    nothing
end

```

The results I got are:

```julia
  1.925835 seconds (2.47 M allocations: 136.644 MiB, 69.99% compilation time)
 15.889349 seconds (42.77 M allocations: 13.568 GiB, 35.68% gc time, 17.75% compilation time)
  7.392910 seconds (13.65 M allocations: 4.117 GiB, 29.45% gc time, 21.90% compilation time)
  2.718504 seconds (2.95 M allocations: 143.125 MiB, 47.44% compilation time)
  2.235974 seconds (3.59 M allocations: 396.477 MiB, 61.13% compilation time)

```

I also don’t understand why case 4 does not lead to the same result as case 1.

Could someone please help me here? I’m guessing I’m messing something up, but have no clue what.

Thanks a lot!

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 11, 2023, 11:32am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/2 "2023-07-11T11:32:27Z")

</div>

Check a profile. What is taking the memory and time? I wouldn’t be surprised if it’s the saving choice and the RNG.

---

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 11, 2023, 3:47pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/3 "2023-07-11T15:47:30Z")

</div>

Thanks a lot for the suggestion!

I’m a noob with profilers, but using ProfileView it seems that a lot of time is spent on `WHITE_NOISE_DIST,` which calls `wiener_randn` and then `randn`. Also some time is spent on copying noise processes at `copy` in `copy_noise_types.jl`

The saving choice is the same for all cases, even the non-problematic ones (1, 3, 5). So saving could only be causing problems in noise processes, correct? Which could explain the `copy` part.

I can also profile memory allocations specifically. Do you have any suggestions for which package to use?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 11, 2023, 4:51pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/4 "2023-07-11T16:51:27Z")

</div>

> [@KalelR](#):
>
> which calls `wiener_randn` and then `randn`

makes sense.

> [@KalelR](#):
>
> Also some time is spent on copying noise processes at `copy` in `copy_noise_types.jl`

Which line?

> [@KalelR](#):
>
> I can also profile memory allocations specifically. Do you have any suggestions for which package to use?

[https://docs.julialang.org/en/v1/manual/profile/#Allocation-Profiler](https://docs.julialang.org/en/v1/manual/profile/#Allocation-Profiler)

---

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 12, 2023, 10:18am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/5 "2023-07-12T10:18:31Z")

</div>

Ok, with

```julia
    N = 40
    noise = 0.1
    ps = [noise]
    u0s = ones(Float64, N)
    tend = 10000.0
    ttrans = 0.0 
    noise_seed = 1
    alg = EM()
    dt = 0.01
    Δt = 10.0
    
    W = WienerProcess(0.0, zeros(length(u0s)))
    prob_independent_noise = SDEProblem(system!, noise!, u0s, (0.0, tend), ps, noise=W)
    Profile.Allocs.clear()
    a = Profile.Allocs.@profile solve(prob_independent_noise, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed);  
    PProf.Allocs.pprof()

```

I got this flame graph:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/1/9/19cf49be138ce4adde537141225a4b832ff28ead.png)

As far as I understood, there are unnecessary allocations. The source for a few of the calls:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/6/8/68530c3a36e4263c962879149a4ebd2bd0b43bc1.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/b/6/b65bfd33e094bf089e15b27553288bca3b96dc30.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/d/edbeab4610b5a57f4732a0d6704a71cdc16df3c9.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/0/80cde0a516a95ab5a7d7e4a71c9a550b58ae9ae5.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/d/8d26517d7502b229044af2ad06b6e1769df11721.png)

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 12, 2023, 1:08pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/6 "2023-07-12T13:08:19Z")

</div>

oh wait, you’re manually constructing a WienerProcess, but not using the in-place form? That’s going to make it a lot slower. Use `WienerProcess!`.

[Classic Noise Processes · DiffEqNoiseProcess.jl](https://docs.sciml.ai/DiffEqNoiseProcess/stable/noise_processes/#DiffEqNoiseProcess.WienerProcess)!

---

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 12, 2023, 1:49pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/7 "2023-07-12T13:49:10Z")

</div>

> Use `WienerProcess!`

I tried the in-place form like this:

```julia
    W = WienerProcess!(0.0, zeros(length(u0s)))
    prob_independent_noise_inplace = SDEProblem(system!, noise!, u0s, (0.0, tend), ps, noise=W)
    sol_inde_in = @time solve(prob_independent_noise_inplace, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed);  

```

(case 3 in my first post). It allocated less, but still a lot. Am I using it wrong?

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 12, 2023, 2:16pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/8 "2023-07-12T14:16:19Z")

</div>

What are the dominant spots in that case? It should be fairly different, since what you were pointing to before were the allocations that this would eliminate.

---

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 12, 2023, 2:32pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/9 "2023-07-12T14:32:47Z")

</div>

Fair, some of them are gone, consistent with a speed up and less allocations. But there are still copy calls:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/2/f2ceee20492477c14b7afa454b7a4ed9605c9611.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/2/9/292c93d4422d1839d645a67837d957a910c4c2da.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/8/6/866202922079be90eade34723f0234a39d45c767.png)

 ![image](https://global.discourse-cdn.com/julialang/original/3X/e/8/e8507fb511531c4d6d05825f3e3d3707baaa57be.png)

Also, to reiterate, the solver runs really nicely without the noise process:

```julia
    prob_independent_noise_2 = SDEProblem(system!, noise!, u0s, (0.0, tend), ps)
    sol_inde_2 = @time solve(prob_independent_noise_2, alg; dt, saveat=ttrans:Δt:tend, seed=noise_seed);   

```

though the results differ. Do you know why?

Thanks again 🙂

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 12, 2023, 6:49pm UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/10 "2023-07-12T18:49:57Z")

</div>

Try `W = WienerProcess!(0.0, zeros(length(u0s)), save_everystep = false)`

---

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 13, 2023, 8:59am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/11 "2023-07-13T08:59:00Z")

</div>

That did it, awesome! Now it’s same speed and memory as solving without the WienerProcess!

One last question: why is the solution different with and without specifying the WienerProcess?

```julia
    prob_1 = SDEProblem(system!, noise!, u0s, (0.0, tend), ps)
    prob_2 = SDEProblem(system!, noise!, u0s, (0.0, tend), ps, noise=W)

```

Are the realizations of the noise just different?

Thanks a lot! 😁

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [July 13, 2023, 10:49am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/12 "2023-07-13T10:49:56Z")

</div>

> [@KalelR](#):
>
> Are the realizations of the noise just different?

That should be all that’s different.

---

<div class="post-metadata">

**Author:** ![KalelR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kalelr/32/43305_2.png) [@KalelR](https://discourse.julialang.org/u/KalelR)\
**Post date:** [July 13, 2023, 10:52am UTC](https://discourse.julialang.org/t/stochastic-differential-equation-with-high-dimensions-is-using-too-much-memory-and-taking-too-long/101478/13 "2023-07-13T10:52:05Z")

</div>

Seems to be. Awesome, thanks once more!