# Tracing memory occupied by existing variables

**URL:** <https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680>\
**Category:** Internals & Design\
**Tags:** memory\
**Created:** [February 2, 2022, 6:47pm UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680 "2022-02-02T18:47:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 2, 2022, 6:47pm UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/1 "2022-02-02T18:47:44Z")

</div>

Hi All,

We are searching for a cause, why our server eventually consumes all memory and crashes, a topic discussed here [Poor performance of garbage collection in multi-threaded application](https://discourse.julialang.org/t/poor-performance-of-garbage-collection-in-multi-threaded-application/75493)  
We have tried to execute GC manually, but it does not seem to help. The effect we observe is very non-deterministic and we do not really know, what is going on.

As a next step, we are considering to try to reach all allocated objects in all modules and compute their size, such that we know, that there is something we have not seen  
I would like to ask, if a code-snippet below make sense?

```julia
function memtracer(m::Module, depth = 0, traced = Set())
	ns = filter(n -> getfield(m, n) isa Module, names(m))
	ns = filter(n -> n ∉ traced, ns)
	traced = union(traced, ns)
	for n in ns
		memtracer(getfield(m, n), depth, traced)
	end

	vars = filter(n -> !(getfield(m, n) isa Module), names(m))
	vars = filter(n -> !(getfield(m, n) isa Function), vars)
	vars = filter(n -> !(getfield(m, n) isa DataType), vars)
	vars = filter(n -> !(getfield(m, n) isa Type), vars)
	isempty(vars) && return()
	println(repeat(" ", depth), "Module: "*string(m))
	mems = map(n -> Base.summarysize(getfield(m, n)), vars)
	stats = sort(collect(zip(vars, mems)), lt = (i,j) -> i[2] < j[2], rev = true)
	l = maximum(length(string(v)) for v in vars) + depth
	for (n,s) in stats
		println(rpad(repeat(" ", depth+4)*string(n), l)," ", s)
	end
	nothing
end

memtracer(Main)

```

Or, of someone else can provide us a different strategy to achieve this, using some nice mechanism we do not know about, we would be very happy.

Thanks to all in advance,  
Tomas

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 2, 2022, 8:52pm UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/2 "2022-02-02T20:52:45Z")

</div>

Are you by any chance reading large data from disk?

---

<div class="post-metadata">

**Author:** ![ianshmean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ianshmean/32/216042_2.png) [@ianshmean](https://discourse.julialang.org/u/ianshmean)\
**Post date:** [February 3, 2022, 2:36am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/3 "2022-02-03T02:36:19Z")

</div>

I think your approach quite similar to this that’s in Base:

```julia
InteractiveUtils.varinfo(Module, recursive = true, all = true, sortby = :size, minsize = 1000)

```

(`minsize` was added in 1.8)

[https://docs.julialang.org/en/v1.8-dev/stdlib/InteractiveUtils/#InteractiveUtils.varinfo](https://docs.julialang.org/en/v1.8-dev/stdlib/InteractiveUtils/#InteractiveUtils.varinfo)

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 3, 2022, 5:40am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/4 "2022-02-03T05:40:49Z")

</div>

I took inspiration there, and did not know that you can write it as this. Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 3, 2022, 5:41am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/5 "2022-02-03T05:41:27Z")

</div>

Yes, we do?  
Is there something what can go wrong there?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [February 3, 2022, 6:04am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/6 "2022-02-03T06:04:42Z")

</div>

I’ve seen this issue popping up from time to time over several years, always related to reading data from files. I reported it here  
[https://github.com/JuliaData/CSV.jl/issues/850](https://github.com/JuliaData/CSV.jl/issues/850)

And investigations led to this issie  
[https://github.com/JuliaLang/julia/issues/42566#issuecomment-950018896](https://github.com/JuliaLang/julia/issues/42566#issuecomment-950018896)

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 3, 2022, 6:16am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/7 "2022-02-03T06:16:54Z")

</div>

Interesting, I will take a look on that. Thanks a lot.  
Very valuable.

---

<div class="post-metadata">

**Author:** ![Tomas\_Pevny](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tomas_pevny/32/25466_2.png) [@Tomas\_Pevny](https://discourse.julialang.org/u/Tomas_Pevny)\
**Post date:** [February 3, 2022, 8:00am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/8 "2022-02-03T08:00:56Z")

</div>

Do you using that using Julia with musl instead of glibc would help?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [February 3, 2022, 9:02am UTC](https://discourse.julialang.org/t/tracing-memory-occupied-by-existing-variables/75680/9 "2022-02-03T09:02:03Z")

</div>

Note that we occasionally have some troubles with musl dynamic loader: [https://github.com/JuliaLang/julia/issues/40556](https://github.com/JuliaLang/julia/issues/40556). But there is a suggested workaround there (it involves recompiling the libc).