Why am I getting OutOfMemoryError() and how do I avoid it?

General Usage
,
1

I am trying to generate combinations of integers in the range from 1 to 12. Using 8 as an argument for the function below I get an OutOfMemory() error. The function works as expected for N < 8. My PC has 32GB of memory of which only 31% is being used. Since binomial(12, 8) == 495 the number of items is small and so is the space occupied by each one. What is happening?

function all_combs(n)
    m12 = 1:12 |> collect
    Base.product(values(repeat([m12], n))...) |> collect |> vec
end

julia> all_combs(8)
ERROR: OutOfMemoryError()

Is there an alternative way to create this array?

2

No issues on my side. Try this
for n = 12

julia> vec(collect(Base.product(values(repeat(collect(1:12), 12))...)))
1-element Vector{NTuple{144, Int64}}:
 (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)

and for n = 8

julia> vec(collect(Base.product(values(repeat(collect(1:12), 8))...)))
1-element Vector{NTuple{96, Int64}}:
 (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
3

You are creating all Combinations with repetition (and order matters):

julia> all_combs(2)
144-element Vector{Tuple{Int64, Int64}}:
(1, 1)
...
(12, 1)
...
(1, 12)
...
(12, 12)

but

binomial(12, 8) == 495

is all combinations without repetition and this is how you can do it:

julia> using Combinatorics

julia> collect(combinations(1:12,8))
495-element Vector{Vector{Int64}}:
...
4

all_combs(n) creates a Vector of length 12^n where each element consists of n Ints (each requiring 8 bytes of memory, assuming a 64-bit system). So for n == 8, you need 12^8 * 8 * 8 bytes, or about 27.5 GB, which is > 69% of your total RAM.

2 Likes
5

The easy answer here is to just not collect and use the iterator directly.

2 Likes