# Memory management in CPLEX.jl
**URL:** https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704
**Category:** Optimization (Mathematical)
**Tags:** question, jump, cplex
**Created:** [February 4, 2024, 4:55pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704 "2024-02-04T16:55:31Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Matheus\_Diogenes\_And](https://discourse.julialang.org/u/Matheus_Diogenes_And)
#### Post date: [February 4, 2024, 4:55pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/1 "2024-02-04T16:55:31Z")
Hello.
I am investigating a behavior related to the execution of [CPLEX.jl](https://github.com/jump-dev/CPLEX.jl). So, let’s consider the following environment:
- Linux pop-os 6.4.6-76060406-generic; And
- Julia version 1.8.4.
And the following commands:
```sh
julia --threads=auto --project=.
```
```julia
julia> include("src/run.jl")
julia> runMyCplexModel("instance_file1") # running a cplex model
julia> runMyCplexModel("instance_file2") # running another cplex model
```
Let’s assume that both, instance 1 (`instance_file1`), and instance 2 (`instance_file2`) lead to huge models, in the number of variables, let’s say something in the order of `40000` variables.
When running the instances, that is
```julia
julia> runMyCplexModel("instance_file1")
julia> runMyCplexModel("instance_file2")
```
linux kills the julia process during the instance 2 (`instance_file2`) execution, due to the amount of consumed RAM memory. However, if I run both instances in different julia sessions, that is
```sh
julia --threads=auto --project=.
```
```julia
julia> runMyCplexModel("instance_file1")
julia> exit()
```
```sh
julia --threads=auto --project=.
```
```julia
julia> runMyCplexModel("instance_file2")
julia> exit()
```
no problem manifests.
I would like to know if there is any julia command to track the amount of allocated RAM memory, so I could provide a more precisest diagnostic.
Thanks and regards.
---
### Author:  [@odow](https://discourse.julialang.org/u/odow)
#### Post date: [February 4, 2024, 7:52pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/2 "2024-02-04T19:52:49Z")
Do you have the code for `runMyCplexModel`? Does it return anything?
You should try
```julia
include("src/run.jl")
runMyCplexModel("instance_file1")
GC.gc() # Force the GC to run
runMyCplexModel("instance_file2")
```
---
### Author:  [@Matheus\_Diogenes\_And](https://discourse.julialang.org/u/Matheus_Diogenes_And)
#### Post date: [February 4, 2024, 8:20pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/3 "2024-02-04T20:20:10Z")
Thanks for the comment.
> Do you have the code for `runMyCplexModel` ? Does it return anything?
Yes, this is very long function, but it is a simple compact MILP formulation as the ones presented at the CPLEX.jl [webpage](https://github.com/JuliaConstraints/CPLEXCP.jl), no callbacks, only the common flow: variables + objective function + and constraints. However, for a model with a bunch of variables.
> ```julia
> include("src/run.jl")
> runMyCplexModel("instance_file1")
> GC.gc() # Force the GC to run
> runMyCplexModel("instance_file2")
>
> ```
Thank you very much, I was thinking about this, but I was unsure about its effectiveness. I think it may solve the issue. I will take the time to replicate the experiment, and then I come back here again.
Regards.
---
### Author:  [@mike\_k](https://discourse.julialang.org/u/mike_k)
#### Post date: [February 5, 2024, 9:18am UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/4 "2024-02-05T09:18:21Z")
On Linux you can track the used memory with the function I have mentioned [here](https://discourse.julialang.org/t/outofmemoryerror-instead-of-allocating-too-much-resources-in-the-job-scheduler/45575/4).
---
### Author:  [@Matheus\_Diogenes\_And](https://discourse.julialang.org/u/Matheus_Diogenes_And)
#### Post date: [February 9, 2024, 1:06pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/5 "2024-02-09T13:06:50Z")
> [@odow](#):
>
> Do you have the code for `runMyCplexModel`? Does it return anything?
>
> You should try
>
> ```julia
> include("src/run.jl")
> runMyCplexModel("instance_file1")
> GC.gc() # Force the GC to run
> runMyCplexModel("instance_file2")
>
> ```
Thanks, somehow it helped, the process is not being killed by the OS anymore. For a thorough investigation on this, I prepared a short example, a TSP instance.
```julia
using JuMP, CPLEX
# https://discourse.julialang.org/t/outofmemoryerror-instead-of-allocating-too-much-resources-in-the-job-scheduler/45575/3
function get_mem_use()
f::IOStream = open( "/proc/self/stat", "r" )
s::AbstractString = read( f, String )
vsize::Int = parse( Int64, split( s )[23] )
mb::Int = Int( ceil( vsize / ( 1024 * 1024 ) ) )
close(f)
return mb::Int
end
#
dist::Matrix{Int} = [ 0 6 9 8 7 3 6 2 3 2 6 6 4 4 5 9 7
6 0 8 3 2 6 8 4 8 8 13 7 5 8 12 10 14
9 8 0 11 10 6 3 9 5 8 4 15 14 13 9 18 9
8 3 11 0 1 7 10 6 10 10 14 6 7 9 14 6 16
7 2 10 1 0 6 9 4 8 9 13 4 6 8 12 8 14
3 6 6 7 6 0 2 3 2 2 7 9 7 7 6 12 8
6 8 3 10 9 2 0 6 2 5 4 12 10 10 6 15 5
2 4 9 6 4 3 6 0 4 4 8 5 4 3 7 8 10
3 8 5 10 8 2 2 4 0 3 4 9 8 7 3 13 6
2 8 8 10 9 2 5 4 3 0 4 6 5 4 3 9 5
6 13 4 14 13 7 4 8 4 4 0 10 9 8 4 13 4
6 7 15 6 4 9 12 5 9 6 10 0 1 3 7 3 10
4 5 14 7 6 7 10 4 8 5 9 1 0 2 6 4 8
4 8 13 9 8 7 10 3 7 4 8 3 2 0 4 5 6
5 12 9 14 12 6 6 7 3 3 4 7 6 4 0 9 2
9 10 18 6 8 12 15 8 13 9 13 3 4 5 9 0 9
7 14 9 16 14 8 5 10 6 5 4 10 8 6 2 9 0 ]
n::Int = size(dist, 1)
depot::Int = 1
#
N::Vector{Int} = 1:n
M::Int = 10000
function myModel()
#
model::Model = direct_model(CPLEX.Optimizer())
set_silent(model)
#
@variable(model, t[N], lower_bound = 0, upper_bound = M)
@variable(model, x[N,N], Bin)
#
@objective(model, Min, sum(dist .* x))
#
@constraint(model, flow[i in N], sum(map(j -> x[i,j], N)) == sum(map(j -> x[j,i], N)))
@constraint(model, visit[i in N], sum(map(j -> x[i,j], N)) == 1)
@constraint(model, t[depot] == 0)
@constraint(model, mtz[i in N, j in setdiff(N, depot)], t[j] >= t[i] + dist[i, j] * x[i, j] - M * (1 - x[i, j]))
@constraint(model, null[i in N], x[i,i] == 0)
#
optimize!(model)
end
println("It 0: ", get_mem_use())
for it::Int in 1:10
myModel()
println("It $it: ", get_mem_use())
GC.gc()
end
```
After running the above snippet, we obtain the following output.
```julia
It 0: 5216
It 1: 6508
It 2: 6509
It 3: 6509
It 4: 6509
It 5: 6509
It 6: 6509
It 7: 6509
It 8: 6509
It 9: 6509
It 10: 6509
```
However, when commenting the line `GC.gc()`, we obtain the following result.
```julia
It 0: 5216
It 1: 6508
It 2: 6511
It 3: 6515
It 4: 6520
It 5: 6524
It 6: 6529
It 7: 6529
It 8: 6529
It 9: 6529
It 10: 6529
```
Not sure if this experiment is isolating properly possible outer noises, if not so, any feedback on this regard would be helpful.
Regards.
---
### Author:  [@odow](https://discourse.julialang.org/u/odow)
#### Post date: [February 9, 2024, 8:25pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/6 "2024-02-09T20:25:01Z")
Hmm i dont know if this is reproducing the original problem.
Is this exactly what your real code is doing? How does your real code store solutions?
---
### Author:  [@Matheus\_Diogenes\_And](https://discourse.julialang.org/u/Matheus_Diogenes_And)
#### Post date: [February 9, 2024, 8:50pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/7 "2024-02-09T20:50:19Z")
Well, my code simply returns the solution (in this case a route). You think that maybe calling `value(x[i,j])` may add some “overhead”?
---
### Author:  [@odow](https://discourse.julialang.org/u/odow)
#### Post date: [February 9, 2024, 9:21pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/8 "2024-02-09T21:21:07Z")
Does it return the model or any decision variables? Or is it just data.
But anyway. If adding the GC call worked then do that. The issue seems to be CPLEX not respecting the total system memory.
You could also try setting parameters like
> **[memory available for working storage](https://www.ibm.com/docs/en/icos/22.1.1?topic=parameters-memory-available-working-storage)**
>
> Specifies an upper limit on the amount of central memory, in megabytes, that CPLEX is permitted to use for working memory.
---
### Author:  [@Matheus\_Diogenes\_And](https://discourse.julialang.org/u/Matheus_Diogenes_And)
#### Post date: [February 11, 2024, 11:35am UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/9 "2024-02-11T11:35:45Z")
Thanks for the comments. I updated the previous example to return the created model.
```julia
using JuMP, CPLEX
# https://discourse.julialang.org/t/outofmemoryerror-instead-of-allocating-too-much-resources-in-the-job-scheduler/45575/3
function get_mem_use()
f::IOStream = open( "/proc/self/stat", "r" )
s::AbstractString = read( f, String )
vsize::Int = parse( Int64, split( s )[23] )
mb::Int = Int( ceil( vsize / ( 1024 * 1024 ) ) )
close(f)
return mb::Int
end
#
dist::Matrix{Int} = [ 0 6 9 8 7 3 6 2 3 2 6 6 4 4 5 9 7
6 0 8 3 2 6 8 4 8 8 13 7 5 8 12 10 14
9 8 0 11 10 6 3 9 5 8 4 15 14 13 9 18 9
8 3 11 0 1 7 10 6 10 10 14 6 7 9 14 6 16
7 2 10 1 0 6 9 4 8 9 13 4 6 8 12 8 14
3 6 6 7 6 0 2 3 2 2 7 9 7 7 6 12 8
6 8 3 10 9 2 0 6 2 5 4 12 10 10 6 15 5
2 4 9 6 4 3 6 0 4 4 8 5 4 3 7 8 10
3 8 5 10 8 2 2 4 0 3 4 9 8 7 3 13 6
2 8 8 10 9 2 5 4 3 0 4 6 5 4 3 9 5
6 13 4 14 13 7 4 8 4 4 0 10 9 8 4 13 4
6 7 15 6 4 9 12 5 9 6 10 0 1 3 7 3 10
4 5 14 7 6 7 10 4 8 5 9 1 0 2 6 4 8
4 8 13 9 8 7 10 3 7 4 8 3 2 0 4 5 6
5 12 9 14 12 6 6 7 3 3 4 7 6 4 0 9 2
9 10 18 6 8 12 15 8 13 9 13 3 4 5 9 0 9
7 14 9 16 14 8 5 10 6 5 4 10 8 6 2 9 0 ]
n::Int = size(dist, 1)
depot::Int = 1
#
N::Vector{Int} = 1:n
M::Int = 10000
function myModel(relax::Bool = false)::Model
#
model::Model = direct_model(CPLEX.Optimizer())
set_silent(model)
#
@variable(model, t[N], lower_bound = 0, upper_bound = M)
if relax
@variable(model, x[N,N], lower_bound = 0, upper_bound = 1)
else
@variable(model, x[N,N], Bin)
end
#
@objective(model, Min, sum(dist .* x))
#
@constraint(model, flow[i in N], sum(map(j -> x[i,j], N)) == sum(map(j -> x[j,i], N)))
@constraint(model, visit[i in N], sum(map(j -> x[i,j], N)) == 1)
@constraint(model, t[depot] == 0)
@constraint(model, mtz[i in N, j in setdiff(N, depot)], t[j] >= t[i] + dist[i, j] * x[i, j] - M * (1 - x[i, j]))
@constraint(model, null[i in N], x[i,i] == 0)
return model
end
function run()
model::Model = myModel(true)
optimize!(model)
model = myModel()
optimize!(model)
end
println("It 0: ", get_mem_use())
for it::Int in 1:10
run()
println("It $it: ", get_mem_use())
# GC.gc()
end
```
And the obtained results are similar to the previous one. Results without `GC.gc()`:
```julia
It 0: 5216
It 1: 6511
It 2: 6520
It 3: 6521
It 4: 6521
It 5: 6525
It 6: 6534
It 7: 6543
It 8: 6552
It 9: 6552
It 10: 6552
```
Results with `GC.gc()`:
```julia
It 0: 5216
It 1: 6511
It 2: 6514
It 3: 6514
It 4: 6514
It 5: 6514
It 6: 6514
It 7: 6514
It 8: 6514
It 9: 6514
It 10: 6514
```
Thanks and regards.
---
### Author:  [@odow](https://discourse.julialang.org/u/odow)
#### Post date: [February 11, 2024, 11:28pm UTC](https://discourse.julialang.org/t/memory-management-in-cplex-jl/109704/10 "2024-02-11T23:28:12Z")
I think this is just a problem with CPLEX. I dont have any suggestions other than GC.gc, unfortunately.