My code running in REPL causes memory usage (as shown in ‘top’) of julia to go up gradually. Calling gc() specifically doesn’y help much. Oftentimes, I have to restart julia.
The code basically reads in a DataFrames via readtable, then it allocates some memory and does some processing, then it may plot some graphs with Plots package, then repeats with a different data file, and so on. Some of the above procedures may be manual, with re-including the code file and re-launching the program function. Even with re-launching the program function, the memory usage still goes up through time. I don’t have global memory allocation, and all allocations are within functions. So I am quite suspicious that it is not my code leaking memory (am I correct in this?).
But anyhow, what can I do to find out about memory leaks?
The problem for me is that memory usage can go very high so the system has to use swap space (on Ubuntu 18.04) at times. And there is no sign that gc kicks in to help. The only solution is to restart julia.
The summarysize of my data structures stay quite constant through the iterations, but the size of julia grows. I even tried to set some large data to “nothing” before quitting the function and then call gc(). All does not help much. Very strange.
Anyone know if this following type of statement can give gc trouble? I have yet done a specific test on it.
Are there any new tools in the horizon to analyse memory usage?
I often find it difficult to fix my memory leaks. As a result my HPC Node kills my scripts at random times due to out of memory. I now write scripts that save their status and just run them in a loop until they have completed their tasks.
But ideally I’d like to fix my leaks. Just setting all my references (that I can identify) to nothing and garbage collecting Doesn’t help much.
Are there any tools available to track down such leaks? Could it be caused by multithreading (I’ve seen suggestions along that on discourse before), the compiler? Me point is I don’t really know where to look.
What kind of precautions do people normally take in Julia when they have to handle very memory intensive tasks? I’m still fairly inexperienced with this so any suggestions will be appreciated.
Pure Julia. Briefly: I’m assembling a large LSQ system, ca 1M x 10K built from a rather complex model and dataset. There is therefore quite a lot of infrastructure that this builds on, which makes it difficult to track down the leak. I had expected that nulling all references would suffice but apparently not.
Because when I restart the script and reload only that data I need (which I thought was all the data I didn’t set to null) the memory usage suddenly goes down by a factor 2-4. (For example)
Maybe this is not technically a leak except in the sense that’s i don’t know where the memory is being used.
I know this is all vague but please believe me when I say that there is no MWE.
Maybe (?) an important piece of information: I heavily use type unstable code (and function barriers for performance critical components.) and understand from a recent thread that this creates a lot of extra work for the Compiler. Could it also lead to higher memory usage and is it possible this left in the system? Would it be worthwhile to experiment with @nospecialize?
P.s: the kind of things I do is to dispatch in values using Val and to use {Any, Any} dictionaries. The need for (or maybe convenience of) this style of coding comes from the fact that I have many different „types“ of data to process.
You mentioned that you are also creating plots (Plots.jl?). Are you using the GR backend? If so, I use that to plot event displays in online reconstructions and also faced memory pile-ups. What you can try is this magic line, this fixed it totally and my code is running now for over a month without memory leaks:
This is not possible without rewriting a fairly large package from scratch. Also - I don’t want to. I like the fact that I can choose to break type stability when it becomes convenient. This gives rise to some very elegant code design.
