Julia GC, heap fragmentation, out of memory, push!/append!

Nice. I didn’t know that. Guess I’m too old school :wink:
Although I deploy on Linux I develop on Windows and there you can still get into the situation where you get to 100% RAM taken and then the julia GC frantically trying to free memory and very large (GBs) fluctuations in julia memory but never stabilizing as long as you keep working (doing requests).

Thanks for clarifying, not sure how often in practice the address space gets (very) fragmented in practice. I may have downplayed the risk too much. I guess there’s a reason Ruby looked into compacting GC (e.g. for long-running web servers), may or may not?) apply less to more typical HPC use of Julia. Anyway there’s a solution with MESH I posted in my latest post. I think you can do that right now with Julia without code changes in Julia, maybe there’s already some benefit (or none). It may need the same simplification of Julia code I mentioned for mimalloc, to see all the free that would happen.

This assumes the OS doesn’t do anything clever with the holes in virtual address space, and I suppose it can’t because they were allocated as memory (maybe used previously), and the OS never informed not free/garbage by now. I suppose it could, and would need help from the libc. Or just allocate very large swap file, it’s cheaper than RAM. Or use MESH.

Incidentally, it’s very likely this issue bit me recently…

Many poople is being hurted by this problem, I am facing the same problem.

I have a computer with RAM 128G, but because of this problem, it turns out that the big RAM is useless.