# Determining size of DataFrame for memory management

**URL:** <https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277>\
**Category:** General Usage\
**Tags:** memory, dataframes\
**Created:** [August 4, 2022, 3:40am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277 "2022-08-04T03:40:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 3:40am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/1 "2022-08-04T03:40:34Z")

</div>

I’m working on some data analysis, and I don’t have as much RAM as I’d ideally like. I’m building a variety of datasets by joining several together etc. I’d like to get a list of all objects in the global environment, filter them by Type, then calculate the size of each dataset, and output a table, so I can manually decide which datasets to delete partway through my analysis…

Any tips?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 3:58am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/2 "2022-08-04T03:58:41Z")

</div>

```julia
help?> varinfo
search: varinfo

  varinfo(m::Module=Main, pattern::Regex=r""; all::Bool = false, imported::Bool = false, sortby::Symbol = :name, minsize::Int = 0)

```

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 3:59am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/3 "2022-08-04T03:59:18Z")

</div>

Useful! Is there a way to make it into a DataFrame or something so I can analyze it programmatically?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:03am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/4 "2022-08-04T04:03:27Z")

</div>

eh, don’t re-invent GC… and you likely can’t clean up variable this way.

if you need to do this kind of gymnastics you probably just want to doing more things lazily / on the fly instead of dumping dataset into in-memory representation (DataFrame)

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:04am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/5 "2022-08-04T04:04:58Z")

</div>

It’s not so much that I want to reinvent GC, I just want to release some variables if they’re too big.

turns out that varinfo() only gives a few hundred megs of variables… Julia is using 12GB of RAM.

GC.gc() doesn’t help ☹

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:06am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/6 "2022-08-04T04:06:55Z")

</div>

you need `recursive=true`.

* * *

> [@dlakelan](#):
>
> GC.gc() doesn’t help ☹

you have leftover reference to these big objects thus GC can’t reclaim them, if you have sample code we might be able to help

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:11am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/7 "2022-08-04T04:11:51Z")

</div>

It’s just hundreds of lines of reading in csv files as datasets, merging them, plotting things, running a few linear regressions, plotting other things… etc etc.

It’s within a jmd file being `include_weaved()`

```julia
varinfo(sortby=:size)
...
  psamh 24.704 MiB 398515×7 DataFrame                                                                                                                                                
  popdata 19.104 MiB 20596×65 DataFrame                                                                                                                                                
  alldat3 4.241 MiB 3965×87 DataFrame                                                                                                                                                 
  alldat2 4.090 MiB 3965×80 DataFrame                                                                                                                                                 
  alldat 3.991 MiB 3965×76 DataFrame                                                                                                                                                 
  bigcount 2.554 MiB 2493×76 DataFrame                                                                                                                                                 
  alldat5 1.460 MiB 1292×91 DataFrame                                                                                                                                                 
  alldat4 1.419 MiB 1292×87 DataFrame                                                                                                                                                 
  whosuic 1.305 MiB 10980×34 DataFrame                                                                                                                                                
  homdata 908.688 KiB 7808×13 DataFrame                                                                                                                                                 
  gdpstack 603.562 KiB 16492×6 DataFrame                                                                                                                                                 
...

```

And yet 8 Gig resident:

 ![image](https://global.discourse-cdn.com/julialang/original/3X/0/9/0947ea246b2f8831126a00df8398065deb19fd27.png)

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:12am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/8 "2022-08-04T04:12:20Z")

</div>

> [@jling](#):
>
> you need `recursive=true`.

where do I need this?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:12am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/9 "2022-08-04T04:12:26Z")

</div>

varinfo

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:13am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/10 "2022-08-04T04:13:26Z")

</div>

doesn’t change anything.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:13am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/11 "2022-08-04T04:13:58Z")

</div>

> [@dlakelan](#):
>
> reading in csv files as datasets

you can read it lazily, or better, write out to Arrow.jl and use mmap to read them.

> [@dlakelan](#):
>
> merging them

try to see if you can merge them “in-place” so you only keep what you absolutely needed.

* * *

have you done a calculation to see if theoretically your data even fits in RAM?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:14am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/12 "2022-08-04T04:14:55Z")

</div>

It’s really not THAT big of data. The biggest dataset is 25MB according to that varinfo.

Some of the data is quite big, but I read in one file at a time, take a small subsample, and vcat them together to form that sampled psamh dataset.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:18am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/13 "2022-08-04T04:18:16Z")

</div>

what about

```julia
varinfo(;all = true, sortby=:size, recursive=true)

```

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:19am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/14 "2022-08-04T04:19:15Z")

</div>

same kind of thing:

````julia
  psamh 24.704 MiB 398515×7 DataFrame                                                                                                                                                
  popdata 19.104 MiB 20596×65 DataFrame                                                                                                                                                
  alldat3 4.241 MiB 3965×87 DataFrame                                                                                                                                                 
  alldat2 4.090 MiB 3965×80 DataFrame                                                                                                                                                 
  alldat 3.991 MiB 3965×76 DataFrame                                                                                                                                                 
  bigcount 2.554 MiB 2493×76 DataFrame                                                                                                                                                 
  alldat5 1.460 MiB 1292×91 DataFrame                                                                                                                                                 
  alldat4 1.419 MiB 1292×87 DataFrame                                                                                                                                                 
  whosuic 1.305 MiB 10980×34 DataFrame                                                                                                                                                
  homdata 908.688 KiB 7808×13 DataFrame                                                                                                                                                 
  gdpstack 603.562 KiB 16492×6 DataFrame ```
````

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:21am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/15 "2022-08-04T04:21:22Z")

</div>

so is this actually a problem then? are you running into OOM? there’s no reason to prematurely free up memory (by OS) if your system memory usage % is low

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:22am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/16 "2022-08-04T04:22:52Z")

</div>

Yeah systemd-oomd is killing codium and all its sub-programs (ie. julia) at the moment Julia is using 54% of my available RAM and swap is up to about 4GB out of 9.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:23am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/17 "2022-08-04T04:23:20Z")

</div>

can you share script and data?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:23am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/18 "2022-08-04T04:23:57Z")

</div>

Yeah, actually it’s on a github, international.jmd over here:

> **[GitHub - dlakelan/GunHomicideResearch: Research on US homicides with...](https://github.com/dlakelan/GunHomicideResearch)**
>
> Research on US homicides with firearms, and related issues like concealed carry, gun prevalence, spatial variability, etc - GitHub - dlakelan/GunHomicideResearch: Research on US homicides with fire...

it’ll download some big datasets including the ACS microdata for the last 5 years …

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 4, 2022, 4:24am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/19 "2022-08-04T04:24:49Z")

</div>

does running them not inside jmd help?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [August 4, 2022, 4:25am UTC](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277/20 "2022-08-04T04:25:24Z")

</div>

good question. I guess I can “tangle()” them and then run that script?

also maybe not inside codium… will have to try that out tomorrow probably.

[Next page](https://discourse.julialang.org/t/determining-size-of-dataframe-for-memory-management/85277.md?page=2)