# Solving EnsembleProblem efficiently for large systems: memory issues

**URL:** <https://discourse.julialang.org/t/solving-ensembleproblem-efficiently-for-large-systems-memory-issues/116146>\
**Category:** Performance\
**Tags:** modelingtoolkit, differentialequation\
**Created:** [June 24, 2024, 2:48pm UTC](https://discourse.julialang.org/t/solving-ensembleproblem-efficiently-for-large-systems-memory-issues/116146 "2024-06-24T14:48:32Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![johannesnauta](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/johannesnauta/32/47434_2.png) [@johannesnauta](https://discourse.julialang.org/u/johannesnauta)\
**Post date:** [July 18, 2024, 11:23am UTC](https://discourse.julialang.org/t/solving-ensembleproblem-efficiently-for-large-systems-memory-issues/116146/9 "2024-07-18T11:23:50Z")

</div>

I have a potential solution that appears to work. The principle is relatively simple

1. Create a dictionary (or any ‘map’) that maps the task ID to an `ODEProblem` ascribed to that specific task. The `ODEProblem` is `deepcopy`d once in the beginning.
2. In the `prob_func`, get the task ID using `current_task()`, `deepcopy` the original `ODEProblem` if it does not exist, otherwise mutate it.

The code looks like this, with some locks in place to prevent multiple tasks/threads writing to the dictionary at the same time:

```julia
#/ Create a dictionary to store task-local problems
tproblems = Dict{Task, ODEProblem}()
tlock = Threads.ReentrantLock()

function set_interactions(prob, k, nrepeats)
    #~ Update prob by setting new interactions and initial conditions
    tid = current_task()
    if !haskey(tproblems, tid)
        Threads.lock(tlock)
        tproblems[tid] = deepcopy(prob)
        Threads.unlock(tlock)
    end
    lprob = tproblems[tid]
    interactionsetter(lprob, amatrices[k])
    statesetter(lprob, rand(S))
    return lprob
end

```

with the `statesetter` and `interactionsetter` as above. This seems to work and indeed uses as much memory as the initial set of deepcopy’s, but nothing more.

If this is **not** the way to go I’d be happy to learn how I should otherwise implement this.

---

_[View the full topic](https://discourse.julialang.org/t/solving-ensembleproblem-efficiently-for-large-systems-memory-issues/116146)._