# Efficient simulation with different path lengths

**URL:** <https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321>\
**Category:** New to Julia\
**Tags:** time-series, monte-carlo, simulations\
**Created:** [June 27, 2024, 5:25pm UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321 "2024-06-27T17:25:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [June 27, 2024, 5:25pm UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321/1 "2024-06-27T17:25:53Z")

</div>

Hi, I’ve enjoyed using Julia on my first research project quite a lot, and I’m now thinking of how to set up my code for my second research project. The computational bottleneck in this project lies in performing many computationally cheap simulations; I would like to make sure that I set this up in an efficient way, so let me begin by explaining what I will need to do:

I will need to simulate N independent time-series. Each of these simulations will consist of an ex-ante unknown number of simulation steps (we repeatedly simulate the time-series from time-step i to (i+1) until some condition is met). For subsequent calculations, I will need to have access to each step of each time-series.

Q1. I assume that initialising a large array/expanding an array to fit the various time-series as they are being simulated is computationally expensive, so I guess it is better to proceed differently? Would something like a list of arrays (each list entry is 1 time series, each array contains the value of the process at the various time-points) be a good idea? Or would it be better to use a list of lists of arrays (each outer list entry is 1 time series, each inner list entry is 1 time-step, each array is the value of the process at this time-step)? Or is yet another idea better?

Q2. I will need to simulate many different combinations of processes, so in my previous project, I’ve been building small simulation blocks for different processes, patching them together as I needed them. This means that several functions (‘layers’) need to be called in sequence to simulate any given process from step i to step i+1. Is this (modulo implementation) fine, or would it be better to somehow ‘compile’ the various ‘layers’ into one function that performs the simulation from step i to step i+1 directly?

I’m curious to hear your thoughts, and of course I can clarify/elaborate where necessary!

---

<div class="post-metadata">

**Author:** ![abraemer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/abraemer/32/51403_2.png) [@abraemer](https://discourse.julialang.org/u/abraemer)\
**Post date:** [June 27, 2024, 5:41pm UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321/2 "2024-06-27T17:41:16Z")

</div>

I think you description is still too vague to really give good recommendations. Could you perhaps provide some MWE/mock-up of the actual structure of the code? It does not necessarily need to be runnable but would be nice. Bonus points if the computations have realistic characteristics.

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [June 27, 2024, 6:26pm UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321/3 "2024-06-27T18:26:01Z")

</div>

Hi, I can certainly type out some code for this tomorrow! For now, let me break out a brief description of the algorithm to see if that conveys my intentions clearly. I’m very open to setting up the code in a way that pays off on the long run: large parts of my old code will need a massive overhaul for this project anyway, so I want to make as good use of it as possible! This is why I don’t have a good piece of code to post as MWE yet!

A typical simulation looks like this, where we have a 2-dimensional process (X,M) to keep track of. We will need to repeat the entire procedure N times.

Step 0: initial values are used, say X[0] = 0 and M[0] = 0.  
Step i + 1: Function 1: Simulate X[i+1] = X[i] + Standard\_normal. Function 2: M[i+1] = M[i] + 1 if X[i+1] \> X[i], M[i] otherwise. When N[i+1] = 10, stop simulating this path.

Effectively, we are simulating a 1-dimensional random walk with gaussian steps, where we track the number of up-steps and stop after stepping up 10 times. As the number of steps taken before reaching 10 up-steps can be arbitrarily large, I foresee problems in initializing an 3-dimensional array to store all my simulations in.

I’ve illustrated that in step i+1, I’m calling two different functions: in another simulation, I might simulate two random walks X and Y, and e.g. let M count how often we have seen X[i] \> Y[i] so far. For ease of reference, I’ll call any such sequential calls to functions to simulate a step ‘layers’; this is what my Q2 refers to.

If necessary/preferred, I’m happy to send a more elaborate example tomorrow, but I hope it gets gist across?

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [June 27, 2024, 7:40pm UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321/4 "2024-06-27T19:40:23Z")

</div>

For Q2: Composing functions from simpler pieces sounds fine, e.g, `f2 ∘ f1` is essentially “compiled into one function” by the Julia compiler.

For Q1: One option might be to hook into the [iterator interface](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-iteration), i.e., effectively making your simulation an iterator of unknown size and rely on `collect` being efficiently implemented for such already. There might be better options though …

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [June 28, 2024, 7:29am UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321/5 "2024-06-28T07:29:01Z")

</div>

Thank you for your suggestions!

For Q1, I didn’t think of setting it up as an iterator yet, this may certainly work, in particular if I can run multiple simulations in parallel and fill up the iterator as it goes. If I get you correctly, you would then get an iterator (walking through simulations) over iterators (walking through steps), right?

One follow-up question: in my analysis I will then need to efficiently access the data, with the typical queries being:

- iterating through simulations, perform some routine at a given step
- extract step n for all simulations that have at least n steps, e.g. to perform some kind of regression.  
I think the former works nicely with an iterator, what are your thoughts on the latter?

---

<div class="post-metadata">

**Author:** ![JADekker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jadekker/32/210281_2.png) [@JADekker](https://discourse.julialang.org/u/JADekker)\
**Post date:** [June 28, 2024, 2:32pm UTC](https://discourse.julialang.org/t/efficient-simulation-with-different-path-lengths/116321/6 "2024-06-28T14:32:25Z")

</div>

Is a more detailed (pseudo-)code or MWE still desirable, or is the outline of my problem clear at this point?