# Memory limitation for large notebooks

**URL:** <https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078>\
**Category:** Pluto\
**Tags:** pluto\
**Created:** [April 14, 2025, 11:00pm UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078 "2025-04-14T23:00:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kirby\_Zhang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kirby_zhang/32/36204_2.png) [@Kirby\_Zhang](https://discourse.julialang.org/u/Kirby_Zhang)\
**Post date:** [April 14, 2025, 11:00pm UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078/1 "2025-04-14T23:00:27Z")

</div>

I’ve been appreciating Pluto for its reactivity and consistency. But it’s occurring to me now that the reactive model may hit limits for large reports. I’m interested in using Pluto for large reports that may involve repeated computations against large datasets. I want each dataset’s memory to be freed once its section has been generated. Is this impossible to do under Pluto’s reactive model? Is Jupyter the only way to create such reports?

---

<div class="post-metadata">

**Author:** ![danielwe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielwe/32/35657_2.png) [@danielwe](https://discourse.julialang.org/u/danielwe)\
**Post date:** [April 14, 2025, 11:07pm UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078/2 "2025-04-14T23:07:06Z")

</div>

Not sure how your datasets map to variables, but you can use functions or `let` blocks to define local scopes. Variables that are local to a scope will be garbage collected as usual after they go out of scope. So, for example, you can write a function that loads the dataset, does all the processing (perhaps by calling other functions defined in other cells), and returns whatever processing results/figures you need, but does _not_ return the variable holding the dataset.

---

<div class="post-metadata">

**Author:** ![Kirby\_Zhang](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kirby_zhang/32/36204_2.png) [@Kirby\_Zhang](https://discourse.julialang.org/u/Kirby_Zhang)\
**Post date:** [April 14, 2025, 11:09pm UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078/3 "2025-04-14T23:09:36Z")

</div>

Got it. I basically have to prevent large data objects from becoming a reactive variable. Limit reactivity to among small objects condensed from the dataset.

---

<div class="post-metadata">

**Author:** ![Vasily\_Pisarev](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vasily_pisarev/32/7929_2.png) [@Vasily\_Pisarev](https://discourse.julialang.org/u/Vasily_Pisarev)\
**Post date:** [April 15, 2025, 1:26pm UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078/4 "2025-04-15T13:26:29Z")

</div>

A bit hacky, but you can also wrap a dataset in a Ref or array and overwrite the object once the computation is done:

```julia
ds_cache = Any[load_dataset()]

let dataset = ds_cache[]
    ...
end

# more cells if needed

ds_cache[] = nothing

```

It will work if you need the loaded dataset to persist through multiple cells. The catch is, `ds_cache[] = nothing` is not guaranteed to run after the computation if you restart the notebook.

---

<div class="post-metadata">

**Author:** ![fonsp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fonsp/32/222349_2.png) [@fonsp](https://discourse.julialang.org/u/fonsp)\
**Post date:** [April 17, 2025, 8:12am UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078/5 "2025-04-17T08:12:12Z")

</div>

Those are some good tips!

Reactive variables in Pluto should also get garbage collected. E.g. a cell with `hello = rand(Bool, 1_000_000)` should allocate, and changing it to `hello = [false, true]` should lead to the old data (1MB) getting freed from memory.

But I can imagine that this is not always working properly, GC is a difficult topic. We don’t have testing for this right now, because I didn’t know how to write a test for this. It would really help if someone could:

1. Find a clear, simple example showing GC not working properly for reactive variables.
2. Write a reliable piece of Julia code that can detect if the object was freed from memory or not. (We could use this in our testing.)

---

<div class="post-metadata">

**Author:** ![danielwe](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/danielwe/32/35657_2.png) [@danielwe](https://discourse.julialang.org/u/danielwe)\
**Post date:** [April 17, 2025, 5:04pm UTC](https://discourse.julialang.org/t/memory-limitation-for-large-notebooks/128078/6 "2025-04-17T17:04:46Z")

</div>

> [@fonsp](#):
>
> Reactive variables in Pluto should also get garbage collected. E.g. a cell with `hello = rand(Bool, 1_000_000)` should allocate, and changing it to `hello = [false, true]` should lead to the old data (1MB) getting freed from memory.

If I understand correctly, this refers to when you _replace_ the definition of a reactive variable by editing and rerunning a cell? You can’t have `hello = rand(Bool, 1_000_000)` in one cell and then `hello = [false, true]` in a later cell, can you?

I understood the OP as being concerned about memory use that remains even in an end-to-end run of a notebook in its “final” report form. In jupyter they’d reuse the same global variable in one cell after the other, for one dataset after the other, allowing the GC to collect old datasets when no longer referenced. With reactivity, you can’t reassign global (reactive) variables, hence the suggestion to only have datasets in local (non-reactive) variables.

In other words, I don’t think there’s any issue with the GC not working properly here.