# Memory build-up when loading DataFrames in a loop

**URL:** <https://discourse.julialang.org/t/memory-build-up-when-loading-dataframes-in-a-loop/109678>\
**Category:** Performance\
**Tags:** question, dataframes\
**Created:** [February 3, 2024, 5:37pm UTC](https://discourse.julialang.org/t/memory-build-up-when-loading-dataframes-in-a-loop/109678 "2024-02-03T17:37:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![isaacjeffersonlee](https://avatars.discourse-cdn.com/v4/letter/i/9e8a1a/32.png) [@isaacjeffersonlee](https://discourse.julialang.org/u/isaacjeffersonlee)\
**Post date:** [February 3, 2024, 5:37pm UTC](https://discourse.julialang.org/t/memory-build-up-when-loading-dataframes-in-a-loop/109678/1 "2024-02-03T17:37:04Z")

</div>

When I sequentially load a very large DataFrame into memory and then sort it in place, the memory usage builds eventually running out of memory?

Is there anything I can do to prevent this? Why is the garbage collector not freeing the memory each loop iteration? My guess would be it is to do with multi-threading?

My use case is that I want to sort some very large parquet files and then re-save them.

MWE:

```julia
using Parquet, DataFrames

fpath = "./example.parquet"
for i in 1:100
    println(i)
    df = DataFrame(read_parquet(fpath))
    sort!(df, [:timestamp])
    empty!(df)
    df = nothing
    GC.gc(true)
end

```

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 3, 2024, 6:11pm UTC](https://discourse.julialang.org/t/memory-build-up-when-loading-dataframes-in-a-loop/109678/2 "2024-02-03T18:11:28Z")

</div>

Is the memory freed only after the loop (I mean for smaller data so that the process does not crash).

In general GC should be able to reclaim memory in the loop you presented (even without calling it explicitly).

---

<div class="post-metadata">

**Author:** ![isaacjeffersonlee](https://avatars.discourse-cdn.com/v4/letter/i/9e8a1a/32.png) [@isaacjeffersonlee](https://discourse.julialang.org/u/isaacjeffersonlee)\
**Post date:** [February 3, 2024, 6:24pm UTC](https://discourse.julialang.org/t/memory-build-up-when-loading-dataframes-in-a-loop/109678/3 "2024-02-03T18:24:02Z")

</div>

No it’s not freed. If I run the script from a REPL for only 5 iterations, the process initially uses 5.2GiB memory, then once the for loop ends and it returns to the REPL the process is using 17.7 GiB  
of memory.

I’m using julia 1.10.0 if that helps.