I have a problem on Julia v0.5 that I need some advice on how to debug. The essential problem is that I have a script that takes several hours to run. As the script is running, the memory usage of each worker slowly grows until the machine runs out of memory and my script is killed. Then I get sad because I wasted several hours.
Without reproducing all of the details, the essential structure of the code looks like:
function do_the_thing() @sync for worker in workers() @async begin input = RemoteChannel() output = RemoteChannel() remotecall(start_worker_loop, worker, input, output, big_matrix_I_only_want_to_move_once) while keep_going() my_input = get_next_input() put!(input, my_input) my_output = take!(output) save_to_file(my_output) end end end end function start_worker_loop(input, output, big_matrix) while true my_input = take!(input) my_output = long_computation(my_input, big_matrix) put!(output, my_output) end end function long_computation(my_input, big_matrix) # lots of math and allocations, but I would expect that all of the allocations I make here # will be garbage collected every time this function returns... end
To reiterate the problem is that the memory usage seems to grow linearly with time (possibly related to the details of what is going on in the
long_computation function), but I don’t even know where to start with debugging this. Any advice or tips would be greatly appreciated.
Thanks for your time!