# Any tips on finding about memory leaks please?

**URL:** <https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019>\
**Category:** General Usage\
**Created:** [August 8, 2018, 5:30am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019 "2018-08-08T05:30:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![LeoK987](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leok987/32/4880_2.png) [@LeoK987](https://discourse.julialang.org/u/LeoK987)\
**Post date:** [August 8, 2018, 5:30am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/1 "2018-08-08T05:30:54Z")

</div>

My code running in REPL causes memory usage (as shown in ‘top’) of julia to go up gradually. Calling gc() specifically doesn’y help much. Oftentimes, I have to restart julia.

The code basically reads in a DataFrames via readtable, then it allocates some memory and does some processing, then it may plot some graphs with Plots package, then repeats with a different data file, and so on. Some of the above procedures may be manual, with re-including the code file and re-launching the program function. Even with re-launching the program function, the memory usage still goes up through time. I don’t have global memory allocation, and all allocations are within functions. So I am quite suspicious that it is not my code leaking memory (am I correct in this?).

But anyhow, what can I do to find out about memory leaks?

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 8, 2018, 5:48am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/2 "2018-08-08T05:48:16Z")

</div>

> [@LeoK987](#):
>
> So I am quite suspicious that it is not my code leaking memory (am I correct in this?)

Are you working in pure Julia, or calling external libraries (eg in C) that allocate memory?

If you are only using Julia, memory usage _per se_ should not be a concern, the garbage collection will take care of it as needed.

---

<div class="post-metadata">

**Author:** ![LeoK987](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leok987/32/4880_2.png) [@LeoK987](https://discourse.julialang.org/u/LeoK987)\
**Post date:** [August 8, 2018, 6:53am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/3 "2018-08-08T06:53:28Z")

</div>

Thanks for replying.

I am using pure Julia (0.6.4), and its packages.

The problem for me is that memory usage can go very high so the system has to use swap space (on Ubuntu 18.04) at times. And there is no sign that gc kicks in to help. The only solution is to restart julia.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [August 8, 2018, 7:33am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/4 "2018-08-08T07:33:13Z")

</div>

Hard to say more without the actual code, but I would first investigate memory usage of objects with `Base.summarysize`. Also see [this question](https://docs.julialang.org/en/latest/manual/faq/#How-do-I-delete-an-object-in-memory?-1).

---

<div class="post-metadata">

**Author:** ![LeoK987](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/leok987/32/4880_2.png) [@LeoK987](https://discourse.julialang.org/u/LeoK987)\
**Post date:** [August 8, 2018, 11:42am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/5 "2018-08-08T11:42:42Z")

</div>

Thanks for the tip.

The `summarysize` of my data structures stay quite constant through the iterations, but the size of julia grows. I even tried to set some large data to “nothing” before quitting the function and then call gc(). All does not help much. Very strange.

Anyone know if this following type of statement can give gc trouble? I have yet done a specific test on it.

`push!(getfield(myData, fields[i]), 0.)`

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [August 26, 2019, 1:10am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/6 "2019-08-26T01:10:12Z")

</div>

Are there any new tools in the horizon to analyse memory usage?

I often find it difficult to fix my memory leaks. As a result my HPC Node kills my scripts at random times due to out of memory. I now write scripts that save their status and just run them in a loop until they have completed their tasks.

But ideally I’d like to fix my leaks. Just setting all my references (that I can identify) to nothing and garbage collecting Doesn’t help much.

Are there any tools available to track down such leaks? Could it be caused by multithreading (I’ve seen suggestions along that on discourse before), the compiler? Me point is I don’t really know where to look.

What kind of precautions do people normally take in Julia when they have to handle very memory intensive tasks? I’m still fairly inexperienced with this so any suggestions will be appreciated.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 26, 2019, 3:47am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/7 "2019-08-26T03:47:56Z")

</div>

does your script use `ccall`s? If it’s pure Julia, you have to be doing something extreme.

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [August 26, 2019, 3:55am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/8 "2019-08-26T03:55:13Z")

</div>

Pure Julia. Briefly: I’m assembling a large LSQ system, ca 1M x 10K built from a rather complex model and dataset. There is therefore quite a lot of infrastructure that this builds on, which makes it difficult to track down the leak. I had expected that nulling all references would suffice but apparently not.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 26, 2019, 4:00am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/9 "2019-08-26T04:00:11Z")

</div>

how do you know it’s a leak? (vs. you actually have that much memory need

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [August 26, 2019, 4:02am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/10 "2019-08-26T04:02:36Z")

</div>

Because when I restart the script and reload only that data I need (which I thought was all the data I didn’t set to null) the memory usage suddenly goes down by a factor 2-4. (For example)

Maybe this is not technically a leak except in the sense that’s i don’t know where the memory is being used.

I know this is all vague but please believe me when I say that there is no MWE.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [August 26, 2019, 4:09am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/11 "2019-08-26T04:09:07Z")

</div>

if you know how much data you need to load, is it possible to pre-allocate? that should prevent memory leak.

---

<div class="post-metadata">

**Author:** ![JeffreySarnoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeffreysarnoff/32/1980_2.png) [@JeffreySarnoff](https://discourse.julialang.org/u/JeffreySarnoff)\
**Post date:** [August 26, 2019, 6:38am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/12 "2019-08-26T06:38:30Z")

</div>

Does this problem exist running on the current release version of Julia?

> **[Download Julia](https://julialang.org/downloads/)**
>
> The official website for the Julia Language. Julia is a language that is fast, dynamic, easy to use, and open source. Click here to learn more.

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [September 1, 2019, 7:43am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/13 "2019-09-01T07:43:04Z")

</div>

I pre-allocate anything I can predict.

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [September 1, 2019, 7:43am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/14 "2019-09-01T07:43:19Z")

</div>

Thank you for the suggestion. I will try this.

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [September 1, 2019, 7:47am UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/15 "2019-09-01T07:47:53Z")

</div>

Maybe (?) an important piece of information: I heavily use type unstable code (and function barriers for performance critical components.) and understand from a recent thread that this creates a lot of extra work for the Compiler. Could it also lead to higher memory usage and is it possible this left in the system? Would it be worthwhile to experiment with @nospecialize?

P.s: the kind of things I do is to dispatch in values using Val and to use {Any, Any} dictionaries. The need for (or maybe convenience of) this style of coding comes from the fact that I have many different „types“ of data to process.

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [September 1, 2019, 3:58pm UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/16 "2019-09-01T15:58:08Z")

</div>

What type unstable code are you heavily using? Rewriting it to be stable night help a ton.

---

<div class="post-metadata">

**Author:** ![tamasgal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamasgal/32/27946_2.png) [@tamasgal](https://discourse.julialang.org/u/tamasgal)\
**Post date:** [September 1, 2019, 4:05pm UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/17 "2019-09-01T16:05:17Z")

</div>

You mentioned that you are also creating plots (`Plots.jl`?). Are you using the `GR` backend? If so, I use that to plot event displays in online reconstructions and also faced memory pile-ups. What you can try is this magic line, this fixed it totally and my code is running now for over a month without memory leaks:

```julia
GR.inline("png")

```

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [September 1, 2019, 6:04pm UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/18 "2019-09-01T18:04:08Z")

</div>

> [@Oscar\_Smith](#):
>
> Rewriting it to be stable night help a ton.

This is not possible without rewriting a fairly large package from scratch. Also - I don’t want to. I like the fact that I can choose to break type stability when it becomes convenient. This gives rise to some very elegant code design.

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [September 1, 2019, 6:04pm UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/19 "2019-09-01T18:04:28Z")

</div>

No, I’m not plotting at all.

---

<div class="post-metadata">

**Author:** ![tamasgal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamasgal/32/27946_2.png) [@tamasgal](https://discourse.julialang.org/u/tamasgal)\
**Post date:** [September 1, 2019, 6:08pm UTC](https://discourse.julialang.org/t/any-tips-on-finding-about-memory-leaks-please/13019/20 "2019-09-01T18:08:59Z")

</div>

Ah sorry I missed that you hijacked this thread 😜