I have a function 1, contain allocation of some small dict, about two dict with less than 10 elements. And I have function 2 call this function 1 in a for loop about 2^12 times. Then I have call function 2 in a for loop for 2^10 times. Then I found my memory increased quickly, but actually I never remember these small dicts.
I tried GC.gc() after each loop, the memory will decrease a lot but still much bigger than my expect.
I don;t know if it’s the small dict couse huge memory.
I want to pre-allocate a dict, so I can use it each time rather than re-allocate a new one. But I have not found a method to pre-allocate a given size dict.
Help! Thanks in advance!
2 Likes
In function 1, first I create a new dict with Dict(), then I append some key-value pairs. How can I use empty! to avoid allocation?
If your dict is small, consider using an alternative implmentation, the ArrayDictionary may be an option?
2 Likes
Thank you, I will try it.
But the biggest problem is my program will be killed because it used lots of memory, about 300G. I don’t why, I didn’t need so much memory to store the needed variables.
I frequently allocate small dict, but I use empty! to avoid re-allocate dict, the memory is still increased quickly.
And I also used a structure of dict’s dict to store sth. I need. I will add some (key, dict) to the dict and read out the (key, dict) pairs throughout my main function. Someone told me that such dict’s dict structure may cause memory leak, but he don’t remember where he saw it, I don’t know if it’s true.
I estimate that the memory I need is about 40G, but the program used up to 300G.