Segmentation fault error with JuMP and Ipopt on HPC cluster

Dear All,

I am trying to solve a large-scale optimization problem that uses JuMP and Ipopt. I get a signal (11): Segmentation fault error trying to run the code on an HPC cluster (details below), however if I switch to a commercial solver such as KNITRO, it works just fine. Also, when I run the same code on my laptop for a smaller problem size, it works with Ipopt too. The code is of the form:

for i in 1:N  # N=50e3

   for j in 1:M # M = 10e3
   
   fun_sol_opt_1(i,j) 
   # sol_opt_1 is a function that uses `JuMP` and `Ipopt`

   fun_sol_opt_2(i,j) 
   # sol_opt_2 is a function that uses `JuMP` and `Ipopt`

  end

end

where I am getting the error:

signal (11): Segmentation fault
in expression starting at /home/optfolder/My_opt_code/run_code.jl:50
dmumps_compre_new_ at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
__dmumps_fac_asm_master_m_MOD_dmumps_fac_asm_niv1 at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
__dmumps_fac_par_m_MOD_dmumps_fac_par at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
dmumps_fac_b_ at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
dmumps_fac_driver_ at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
dmumps_ at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
dmumps_f77_ at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
dmumps_c at /home/optfolder/.julia/artifacts/9d7c145c183adb9f2f4df30a7531e4ef1c95cdc3/lib/libdmumps.so (unknown line)
_ZN5Ipopt20MumpsSolverInterface13FactorizationEbi at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt20MumpsSolverInterface10MultiSolveEbPKiS2_iPdbi at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt16TSymLinearSolver10MultiSolveERKNS_9SymMatrixERSt6vectorINS_8SmartPtrIKNS_6VectorEEESaIS8_EERS4_INS5_IS6_EESaISC_EEbi at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt18StdAugSystemSolver10MultiSolveEPKNS_9SymMatrixEdPKNS_6VectorEdS6_dPKNS_6MatrixES6_dS9_S6_dRSt6vectorINS_8SmartPtrIS5_EESaISC_EESF_SF_SF_RSA_INSB_IS4_EESaISG_EESJ_SJ_SJ_bi at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt15AugSystemSolver5SolveEPKNS_9SymMatrixEdPKNS_6VectorEdS6_dPKNS_6MatrixES6_dS9_S6_dRS5_SA_SA_SA_RS4_SB_SB_SB_bi at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt17PDFullSpaceSolver9SolveOnceEbbRKNS_9SymMatrixERKNS_6MatrixES6_S6_S6_S6_S6_RKNS_6VectorES9_S9_S9_S9_S9_S9_S9_S9_S9_ddRKNS_14IteratesVectorERSA_ at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt17PDFullSpaceSolver5SolveEddRKNS_14IteratesVectorERS1_bb at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt21PDSearchDirCalculator22ComputeSearchDirectionEv at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt14IpoptAlgorithm22ComputeSearchDirectionEv at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt14IpoptAlgorithm8OptimizeEb at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication13call_optimizeEv at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication11OptimizeNLPERKNS_8SmartPtrINS_3NLPEEERNS1_INS_16AlgorithmBuilderEEE at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication11OptimizeNLPERKNS_8SmartPtrINS_3NLPEEE at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
_ZN5Ipopt16IpoptApplication12OptimizeTNLPERKNS_8SmartPtrINS_4TNLPEEE at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
IpoptSolve at /home/optfolder/.julia/artifacts/04cddd7adc69260fa855b10afdbfbc8c244c7862/lib/libipopt.so (unknown line)
solveProblem at /home/optfolder/.julia/packages/Ipopt/vtrOr/src/Ipopt.jl:532
optimize! at /home/optfolder/.julia/packages/Ipopt/vtrOr/src/MOI_wrapper.jl:1713
optimize! at /home/optfolder/.julia/packages/MathOptInterface/YDdD3/src/Bridges/bridge_optimizer.jl:319
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
optimize! at /home/optfolder/.julia/packages/MathOptInterface/YDdD3/src/Utilities/cachingoptimizer.jl:252
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
#optimize!#122 at /home/optfolder/.julia/packages/JuMP/Xrr7O/src/optimizer_interface.jl:185
optimize! at /home/optfolder/.julia/packages/JuMP/Xrr7O/src/optimizer_interface.jl:157 [inlined]
optimize! at /home/optfolder/.julia/packages/JuMP/Xrr7O/src/optimizer_interface.jl:157 [inlined]
#BnB_PEP_solver_one_stepsize#27 at /home/optfolder/My_opt_code/files_with_functions.jl:1463
BnB_PEP_solver_one_stepsize##kw at /home/optfolder/My_opt_code/files_with_functions.jl:940
unknown function (ip: 0x7f8295cfa9f3)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
top-level scope at /home/optfolder/My_opt_code/run_code.jl:79
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:834
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:913
jl_load_rewrite at /buildworker/worker/package_linux64/build/src/toplevel.c:914
include at ./Base.jl:380
include at ./Base.jl:368
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
exec_options at ./client.jl:296
_start at ./client.jl:506
jfptr__start_60376.clone_1 at /state/partition1/llgrid/pkg/julia/julia-1.5.2/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
unknown function (ip: 0x401931)
unknown function (ip: 0x401533)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 216423358 (Pool: 216196085; Big: 227273); GC: 149
/var/spool/slurmd/job10668471/slurm_script: line 17: 21383 Segmentation fault 

Is there any reason why I am getting this error? Any tips regarding how to avoid it would be much appreciated.

The current version of Ipopt with MUMPS is not thread-safe. Use a different linear solver.

Great, will do! Thanks @odow !