Is there any way to defragment memory if it’s already fragmented, or is it tied to how large chunks of memory and where in the code do I allocate them?
It is more of an intrinsic problem. In the (g)olden days they used compacting garbage collection to fight fragmentation, but this is not an option here AFAIU (or are we?). You’ll probably have to identify the critical paths of your application and use an adequate preallocation or arena allocation strategy.
Edit: just looked it up, here another link.
No and yes.
1 Like
FYI there’s “compacting malloc”
which can be used without re-building julia (just LD_PRELOAD) IIUC.
Julia’s GC manages pool on its own so I don’t know how effective it is. But they also showed it was effective with Ruby.
(Edit: It’s likely irrelevant to this issue. Their Ruby example is based on that large string allocation in Ruby directly calls malloc.)
2 Likes