Nested loops are consuming a lot of memory

General Usage
,
1

I have a nested loop iteration scheme that’s taking up a lot of memory. I do understand that I should not have globals but even after I included everything in a function, the memory situation didn’t improve. It just accumulates after each iteration as if there is no garbage collection.

Here is a workable example that’s similar to my code.

I have two files. First, functions.jl:

##functions.jl
module functions

function getMatrix(A)
    L = rand(A,A);
    return L;
end

function loopOne(A, B)
    res = 0;
    for i = 1:B
        res = inv(getMatrix(A));
    end
    return(res);
end

end

Second file main.jl:

##main.jl
include("functions.jl")
function main(C)
    A = 50;
    B = 30;
    res = 0;
    for i =1:C
        mat = functions.loopOne(A,B);
        res = mat .+ 1;
    end
    return res;
end
main(100)

When I execute julia main.jl, the memory increases as I extend the loop in main(C) (sometimes to more than millions of allocations and 10GiBs if I increase C to 10,000).

Issue: I want to reduce the memory consumption. Since I’m not creating any new arrays throughout the loop, the consumption of memory should be pretty much close to constant regardless of how much I increase C. However, that’s not the case here. The variables that I have are just L, A, B, res, mat. My code just updates them throughout. I don’t understand why it’s creating new arrays.

What I want: I’m assigning new values to variables with the same names, as the previously assigned values are useless when the loop enters into a new iteration. Therefore, I want the memory usage to stay constant.

I know that the example looks useless but it resembles the structure that I have. Can someone please help? Thank you.

2

Could you update your example so that it actually runs? Right now, functions.getMatrix and functions.loopOne are undefined, and the res that you’re trying to return in loopOne isn’t in scope.

1 Like
3

I’d suggest posting your actual code. That way people could see which new array allocation is needed or not.
But I don’t know if get your problem right. Are you worried about unnecessary allocation or garbage collection not being triggered?

4

@favba @tkoolen

I updated the code so this should work. The issue is that as I increase C in main(C), the memory consumption increases. How can I prevent this from happening? I’m not creating any new arrays throughout the loop so intuitively this shouldn’t happen. I don’t have any globals either.

5

I’m confused; you create a lot of new arrays so naturally the number of allocations become larger with each loop? Also, what version are you on? Your example still does not run on 0.6.

6

@fredrikekre

Fixed the code again. This time it should work.

I’m using 0.6. Why are there new arrays? The variables that I have are just L, A, B, res, mat ? My code just updates them throughout.

7

You’ve also posted this over at StackOverflow, where it has attracted a nice answer. Please don’t simultaneously post the same question in multiple places — it splits the discussion and duplicates the efforts of many helpful volunteers.

5 Likes