I had a similar-looking problem where memory use seemed to grow linearly with time. I managed to simplify the code to the point where I could reproduce the “memory leak” with a 3-line for loop. This was instrumental in helping me identify the problem.
Linking my solution on the off-chance that it’s relevant.