Memory management in CPLEX.jl

Specific Domains Optimization (Mathematical)
, ,
1

Hello.

I am investigating a behavior related to the execution of CPLEX.jl. So, let’s consider the following environment:

  • Linux pop-os 6.4.6-76060406-generic; And
  • Julia version 1.8.4.

And the following commands:

julia --threads=auto --project=.

julia> include("src/run.jl")
julia> runMyCplexModel("instance_file1") # running a cplex model
julia> runMyCplexModel("instance_file2") # running another cplex model

Let’s assume that both, instance 1 (instance_file1), and instance 2 (instance_file2) lead to huge models, in the number of variables, let’s say something in the order of 40000 variables.

When running the instances, that is

julia> runMyCplexModel("instance_file1") 
julia> runMyCplexModel("instance_file2")

linux kills the julia process during the instance 2 (instance_file2) execution, due to the amount of consumed RAM memory. However, if I run both instances in different julia sessions, that is

julia --threads=auto --project=.

julia> runMyCplexModel("instance_file1") 
julia> exit()

julia --threads=auto --project=.

julia> runMyCplexModel("instance_file2") 
julia> exit()

no problem manifests.

I would like to know if there is any julia command to track the amount of allocated RAM memory, so I could provide a more precisest diagnostic.

Thanks and regards.

2

Do you have the code for runMyCplexModel? Does it return anything?

You should try

include("src/run.jl")
runMyCplexModel("instance_file1")
GC.gc()  # Force the GC to run
runMyCplexModel("instance_file2")
1 Like
3

Thanks for the comment.

Do you have the code for runMyCplexModel ? Does it return anything?

Yes, this is very long function, but it is a simple compact MILP formulation as the ones presented at the CPLEX.jl webpage, no callbacks, only the common flow: variables + objective function + and constraints. However, for a model with a bunch of variables.

include("src/run.jl")
runMyCplexModel("instance_file1")
GC.gc()  # Force the GC to run
runMyCplexModel("instance_file2")

Thank you very much, I was thinking about this, but I was unsure about its effectiveness. I think it may solve the issue. I will take the time to replicate the experiment, and then I come back here again.

Regards.

4

On Linux you can track the used memory with the function I have mentioned here.

2 Likes
5

Thanks, somehow it helped, the process is not being killed by the OS anymore. For a thorough investigation on this, I prepared a short example, a TSP instance.

using JuMP, CPLEX

# https://discourse.julialang.org/t/outofmemoryerror-instead-of-allocating-too-much-resources-in-the-job-scheduler/45575/3
function get_mem_use()
    f::IOStream         = open( "/proc/self/stat", "r" )
    s::AbstractString   = read( f, String )
    vsize::Int          = parse( Int64, split( s )[23] )
    mb::Int             = Int( ceil( vsize / ( 1024 * 1024 ) ) )
    close(f)
    return mb::Int
end

#
dist::Matrix{Int} = [ 0 6 9 8 7 3 6 2 3 2 6 6 4 4 5 9 7
                     6 0 8 3 2 6 8 4 8 8 13 7 5 8 12 10 14
                     9 8 0 11 10 6 3 9 5 8 4 15 14 13 9 18 9
                     8 3 11 0 1 7 10 6 10 10 14 6 7 9 14 6 16
                     7 2 10 1 0 6 9 4 8 9 13 4 6 8 12 8 14
                     3 6 6 7 6 0 2 3 2 2 7 9 7 7 6 12 8
                     6 8 3 10 9 2 0 6 2 5 4 12 10 10 6 15 5
                     2 4 9 6 4 3 6 0 4 4 8 5 4 3 7 8 10
                     3 8 5 10 8 2 2 4 0 3 4 9 8 7 3 13 6
                     2 8 8 10 9 2 5 4 3 0 4 6 5 4 3 9 5
                     6 13 4 14 13 7 4 8 4 4 0 10 9 8 4 13 4
                     6 7 15 6 4 9 12 5 9 6 10 0 1 3 7 3 10
                     4 5 14 7 6 7 10 4 8 5 9 1 0 2 6 4 8
                     4 8 13 9 8 7 10 3 7 4 8 3 2 0 4 5 6
                     5 12 9 14 12 6 6 7 3 3 4 7 6 4 0 9 2
                     9 10 18 6 8 12 15 8 13 9 13 3 4 5 9 0 9
                     7 14 9 16 14 8 5 10 6 5 4 10 8 6 2 9 0 ]

n::Int = size(dist, 1)
depot::Int = 1

#
N::Vector{Int} = 1:n
M::Int = 10000

function myModel()

    #
    model::Model = direct_model(CPLEX.Optimizer())
    set_silent(model)

    #
    @variable(model, t[N], lower_bound = 0, upper_bound = M)
    @variable(model, x[N,N], Bin)

    #
    @objective(model, Min, sum(dist .* x))

    #
    @constraint(model, flow[i in N], sum(map(j -> x[i,j], N)) == sum(map(j -> x[j,i], N)))
    @constraint(model, visit[i in N], sum(map(j -> x[i,j], N)) == 1)
    @constraint(model, t[depot] == 0)
    @constraint(model, mtz[i in N, j in setdiff(N, depot)], t[j] >= t[i] + dist[i, j] * x[i, j] - M * (1 - x[i, j]))
    @constraint(model, null[i in N], x[i,i] == 0)

    #
    optimize!(model)

end

println("It 0: ", get_mem_use())

for it::Int in 1:10

    myModel()
    println("It $it: ", get_mem_use())
    GC.gc()

end

After running the above snippet, we obtain the following output.

It 0: 5216
It 1: 6508
It 2: 6509
It 3: 6509
It 4: 6509
It 5: 6509
It 6: 6509
It 7: 6509
It 8: 6509
It 9: 6509
It 10: 6509

However, when commenting the line GC.gc(), we obtain the following result.

It 0: 5216
It 1: 6508
It 2: 6511
It 3: 6515
It 4: 6520
It 5: 6524
It 6: 6529
It 7: 6529
It 8: 6529
It 9: 6529
It 10: 6529

Not sure if this experiment is isolating properly possible outer noises, if not so, any feedback on this regard would be helpful.

Regards.

6

Hmm i dont know if this is reproducing the original problem.

Is this exactly what your real code is doing? How does your real code store solutions?

1 Like
7

Well, my code simply returns the solution (in this case a route). You think that maybe calling value(x[i,j]) may add some “overhead”?

8

Does it return the model or any decision variables? Or is it just data.

But anyway. If adding the GC call worked then do that. The issue seems to be CPLEX not respecting the total system memory.

You could also try setting parameters like

1 Like
9

Thanks for the comments. I updated the previous example to return the created model.

using JuMP, CPLEX

# https://discourse.julialang.org/t/outofmemoryerror-instead-of-allocating-too-much-resources-in-the-job-scheduler/45575/3
function get_mem_use()
    f::IOStream         = open( "/proc/self/stat", "r" )
    s::AbstractString   = read( f, String )
    vsize::Int          = parse( Int64, split( s )[23] )
    mb::Int             = Int( ceil( vsize / ( 1024 * 1024 ) ) )
    close(f)
    return mb::Int
end

#
dist::Matrix{Int} = [ 0 6 9 8 7 3 6 2 3 2 6 6 4 4 5 9 7
                     6 0 8 3 2 6 8 4 8 8 13 7 5 8 12 10 14
                     9 8 0 11 10 6 3 9 5 8 4 15 14 13 9 18 9
                     8 3 11 0 1 7 10 6 10 10 14 6 7 9 14 6 16
                     7 2 10 1 0 6 9 4 8 9 13 4 6 8 12 8 14
                     3 6 6 7 6 0 2 3 2 2 7 9 7 7 6 12 8
                     6 8 3 10 9 2 0 6 2 5 4 12 10 10 6 15 5
                     2 4 9 6 4 3 6 0 4 4 8 5 4 3 7 8 10
                     3 8 5 10 8 2 2 4 0 3 4 9 8 7 3 13 6
                     2 8 8 10 9 2 5 4 3 0 4 6 5 4 3 9 5
                     6 13 4 14 13 7 4 8 4 4 0 10 9 8 4 13 4
                     6 7 15 6 4 9 12 5 9 6 10 0 1 3 7 3 10
                     4 5 14 7 6 7 10 4 8 5 9 1 0 2 6 4 8
                     4 8 13 9 8 7 10 3 7 4 8 3 2 0 4 5 6
                     5 12 9 14 12 6 6 7 3 3 4 7 6 4 0 9 2
                     9 10 18 6 8 12 15 8 13 9 13 3 4 5 9 0 9
                     7 14 9 16 14 8 5 10 6 5 4 10 8 6 2 9 0 ]

n::Int = size(dist, 1)
depot::Int = 1

#
N::Vector{Int} = 1:n
M::Int = 10000

function myModel(relax::Bool = false)::Model

    #
    model::Model = direct_model(CPLEX.Optimizer())
    set_silent(model)

    #
    @variable(model, t[N], lower_bound = 0, upper_bound = M)
    if relax
        @variable(model, x[N,N], lower_bound = 0, upper_bound = 1)
    else
        @variable(model, x[N,N], Bin)
    end

    #
    @objective(model, Min, sum(dist .* x))

    #
    @constraint(model, flow[i in N], sum(map(j -> x[i,j], N)) == sum(map(j -> x[j,i], N)))
    @constraint(model, visit[i in N], sum(map(j -> x[i,j], N)) == 1)
    @constraint(model, t[depot] == 0)
    @constraint(model, mtz[i in N, j in setdiff(N, depot)], t[j] >= t[i] + dist[i, j] * x[i, j] - M * (1 - x[i, j]))
    @constraint(model, null[i in N], x[i,i] == 0)

    return model


end

function run()

    model::Model = myModel(true)
    optimize!(model)

    model = myModel()
    optimize!(model)
end

println("It 0: ", get_mem_use())

for it::Int in 1:10

    run()
    println("It $it: ", get_mem_use())
#    GC.gc()

end

And the obtained results are similar to the previous one. Results without GC.gc():

It 0: 5216
It 1: 6511
It 2: 6520
It 3: 6521
It 4: 6521
It 5: 6525
It 6: 6534
It 7: 6543
It 8: 6552
It 9: 6552
It 10: 6552

Results with GC.gc():

It 0: 5216
It 1: 6511
It 2: 6514
It 3: 6514
It 4: 6514
It 5: 6514
It 6: 6514
It 7: 6514
It 8: 6514
It 9: 6514
It 10: 6514

Thanks and regards.

10

I think this is just a problem with CPLEX. I dont have any suggestions other than GC.gc, unfortunately.

1 Like