Memory Debugging

[ilker]

I implemented a complex neural net by using Knet/AutoGrad. Then, I realized that RAM usage of my experiment gets bigger and bigger slowly in the training even though it was a GPU experiment. In simpler models, I don’t have this kind of problem (constant, consistent memory usage after a while). I detected the section that causes the problem, but I still need to debug it in order to find what I do wrong and fix it. I tried the following so far (but nothing yet),

• call gc() in somewhere relevant
• used valgrind to check if there are any leakage
• call whos() periodically and analyze memory usage

I just need some idea or tool for general debugging this kind of memory issues in Julia. Thanks a lot!

[Michael_Eastwood]

I had a similar-looking problem where memory use seemed to grow linearly with time. I managed to simplify the code to the point where I could reproduce the “memory leak” with a 3-line for loop. This was instrumental in helping me identify the problem.

Linking my solution on the off-chance that it’s relevant.

[Solved] Debugging an apparent memory leak

Hello all, I have a problem on Julia v0.5 that I need some advice on how to debug. The essential problem is that I have a script that takes several hours to run. As the script is running, the memory usage of each worker slowly grows until the machine runs out of memory and my script is killed. Then I get sad because I wasted several hours. Without reproducing all of the details, the essential structure of the code looks like: function do_the_thing() @sync for worker in workers() …