I have been using Pardiso.jl 0.5.4 to solve large linear systems of equations with many RHS (80k x 80k). From time to time depending on the size of the system and wether Julia calls GarbageCollect then it will create a segmentation fault and do a core dump.
*** Error in `julia': free(): invalid pointer: 0x00002ae2d4251040 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x2ac1d8666679]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x96346)[0x2ac1d9411346]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_gc_collect+0x28b)[0x2ac1d9411e5b]
[0x2ac301df9195]
[0x2ac301df91b0]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_apply_generic+0x27e)[0x2ac1d93ba26e]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x596a5)[0x2ac1d93d46a5]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x5909c)[0x2ac1d93d409c]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x59e93)[0x2ac1d93d4e93]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x5ac28)[0x2ac1d93d5c28]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x752dc)[0x2ac1d93f02dc]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0x75b5c)[0x2ac1d93f0b5c]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_toplevel_eval_in+0xaa)[0x2ac1d93f1eda]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x72345c)[0x2ac1ead3b45c]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x723e8a)[0x2ac1ead3be8a]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x723f97)[0x2ac1ead3bf97]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x724317)[0x2ac1ead3c317]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_apply_generic+0x27e)[0x2ac1d93ba26e]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x80fe35)[0x2ac1eae27e35]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x810563)[0x2ac1eae28563]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x8105a0)[0x2ac1eae285a0]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_apply_generic+0x27e)[0x2ac1d93ba26e]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x79ce03)[0x2ac1eadb4e03]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x79ce4c)[0x2ac1eadb4e4c]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_apply_generic+0x27e)[0x2ac1d93ba26e]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(jl_f__call_latest+0x39)[0x2ac1d93c75f9]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x1496e)[0x2ac1ea62c96e]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x1d089)[0x2ac1ea635089]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x1d626)[0x2ac1ea635626]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so(+0x1d729)[0x2ac1ea635729]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(ijl_apply_generic+0x27e)[0x2ac1d93ba26e]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(+0xa01f4)[0x2ac1d941b1f4]
/nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libjulia-internal.so.1(jl_repl_entrypoint+0x8a)[0x2ac1d941bc5a]
julia(main+0x9)[0x400bd9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ac1d8607505]
julia[0x400c09]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:2a 6685389495 /nopt/nrel/apps/julia/julia-1.9.3/usr/bin/julia
00601000-00602000 r--p 00001000 00:2a 6685389495 /nopt/nrel/apps/julia/julia-1.9.3/usr/bin/julia
00602000-00603000 rw-p 00002000 00:2a 6685389495 /nopt/nrel/apps/julia/julia-1.9.3/usr/bin/julia
01dbd000-13787000 rw-p 00000000 00:00 0 [heap]
2ac1d7d70000-2ac1d7d92000 r-xp 00000000 00:14 144267 /usr/lib64/ld-2.17.so
2ac1d7d92000-2ac1d7d94000 rw-p 00000000 00:00 0
2ac1d7d94000-2ac1d7d95000 r--p 00000000 00:00 0
2ac1d7d95000-2ac1d7d97000 ---p 00000000 00:00 0
2ac1d7d97000-2ac1d7d99000 rw-p 00000000 00:00 0
2ac1d7d99000-2ac1d7da0000 r--s 00000000 00:14 146705 /usr/lib64/gconv/gconv-modules.cache
2ac1d7da0000-2ac1d7da1000 r--p 00000000 00:2a 6683467224 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libssp.so.0.0.0
2ac1d7da1000-2ac1d7da2000 r-xp 00001000 00:2a 6683467224 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libssp.so.0.0.0
2ac1d7da2000-2ac1d7da3000 r--p 00002000 00:2a 6683467224 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libssp.so.0.0.0
2ac1d7da3000-2ac1d7da4000 r--p 00002000 00:2a 6683467224 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libssp.so.0.0.0
2ac1d7da4000-2ac1d7da5000 rw-p 00003000 00:2a 6683467224 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libssp.so.0.0.0
2ac1d7da5000-2ac1d7da9000 rw-p 00000000 00:00 0
2ac1d7da9000-2ac1d7dad000 r--p 00000000 00:2a 6683467203 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libgcc_s.so.1
2ac1d7dad000-2ac1d7dc4000 r-xp 00004000 00:2a 6683467203 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libgcc_s.so.1
2ac1d7dc4000-2ac1d7dc8000 r--p 0001b000 00:2a 6683467203 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libgcc_s.so.1
2ac1d7dc8000-2ac1d7dc9000 r--p 0001e000 00:2a 6683467203 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libgcc_s.so.1
2ac1d7dc9000-2ac1d7dca000 rw-p 0001f000 00:2a 6683467203 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libgcc_s.so.1
2ac1d7dca000-2ac1d7dcc000 r--p 00000000 00:2a 6683467202 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libatomic.so.1.2.0
2ac1d7dcc000-2ac1d7dcf000 r-xp 00002000 00:2a 6683467202 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libatomic.so.1.2.0
2ac1d7dcf000-2ac1d7dd1000 r--p 00005000 00:2a 6683467202 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libatomic.so.1.2.0
2ac1d7dd1000-2ac1d7dd2000 r--p 00006000 00:2a 6683467202 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libatomic.so.1.2.0
2ac1d7dd2000-2ac1d7dd3000 rw-p 00007000 00:2a 6683467202 /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/libatomic.so.1.2.0
2ac1d7dd3000-2ac1d7f1f000 rw-p 00000000 00:00 0
2ac1d7f1f000-2ac1d7f2f000 rwxp 00000000 00:00 0
2ac1d7f2f000-2ac1d7f30000 r--p 00000000 00:2a 6780259824 /home/jlara/.julia/compiled/v1.9/DelimitedFiles/dlKZm_B3g8R.so
2ac1d7f30000-2ac1d7f31000 r-xp 00000000 00:2a 6780259824 /home/jlara/.julia/compiled/v1.9/DelimitedFiles/dlKZm_B3g8R.so
2ac1d7f31000-2ac1d7f32000 r--p 00000000 00:2a 6780259824 /home/jlara/.julia/compiled/v1.9/DelimitedFiles/dlKZm_B3g8R.so
2ac1d7f32000-2ac1d7f58000 rw-p 00000000 00:2a 6780259824 /home/jlara/.julia/compiled/v1.9/DelimitedFiles/dlKZm_B3g8R.so
2ac1d7f58000-2ac1d7f5c000 rw-p 00000000 00:00 0
2ac1d7f5c000-2ac1d7f7d000 rw-p 00000000 00:00 0
2ac1d7f7d000-2ac1d7f7e000 r--p 00000000 00:2a 6799758597 /home/jlara/.julia/compiled/v1.9/PrecompileTools/AQ9Mk_B3g8R.so
2ac1d7f7e000-2ac1d7f7f000 r-xp 00000000 00:2a 6799758597 /home/jlara/.julia/compiled/v1.9/PrecompileTools/AQ9Mk_B3g8R.so
2ac1d7f7f000-2ac1d7f80000 r--p 00000000 00:2a 6799758597 /home/jlara/.julia/compiled/v1.9/PrecompileTools/AQ9Mk_B3g8R.so
2ac1d7f80000-2ac1d7f8c000 rw-p 00000000 00:2a 6799758597 /home/jlara/.julia/compiled/v1.9/PrecompileTools/AQ9Mk_B3g8R.so
2ac1d7f91000-2ac1d7f92000 r--p 00021000 00:14 144267 /usr/lib64/ld-2.17.so
2ac1d7f92000-2ac1d7f93000 rw-p 00022000 00:14 144267 /usr/lib64/ld-2.17.so
2ac1d7f93000-2ac1d7f94000 rw-p 00000000 00:00 0
2ac1d7f94000-2ac1d7fc3000 r-xp 00000000 00:2a 4387201206 /nopt/xalt/2.7.3/lib64/libxalt_init.so
2ac1d7fc3000-2ac1d81c2000 ---p 0002f000 00:2a 4387201206 /nopt/xalt/2.7.3/lib64/libxalt_init.so
2ac1d81c2000-2ac1d81c3000 r--p 0002e000 00:2a 4387201206 /nopt/xalt/2.7.3/lib64/libxalt_init.so
2ac1d81c3000-2ac1d81c4000 rw-p 0002f000 00:2a 4387201206 /nopt/xalt/2.7.3/lib64/libxalt_init.so
2ac1d81c4000-2ac1d81c5000 rw-p 00000000 00:00 0
2ac1d81c5000-2ac1d81c7000 r-xp 00000000 00:14 146569 /usr/lib64/libdl-2.17.so
2ac1d81c7000-2ac1d83c7000 ---p 00002000 00:14 146569 /usr/lib64/libdl-2.17.so
2ac1d83c7000-2ac1d83c8000 r--p 00002000 00:14 146569 /usr/lib64/libdl-2.17.so
[30401] signal (6.-6): Aborted
in expression starting at REPL[194]:1
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__libc_message at /lib64/libc.so.6 (unknown line)
_int_free at /lib64/libc.so.6 (unknown line)
jl_gc_free_array at /nopt/nrel/apps/julia/julia-1.9.3/src/gc.c:1353 [inlined]
sweep_malloced_arrays at /nopt/nrel/apps/julia/julia-1.9.3/src/gc.c:1378 [inlined]
gc_sweep_other at /nopt/nrel/apps/julia/julia-1.9.3/src/gc.c:1736 [inlined]
_jl_gc_collect at /nopt/nrel/apps/julia/julia-1.9.3/src/gc.c:3544
ijl_gc_collect at /nopt/nrel/apps/julia/julia-1.9.3/src/gc.c:3713
gc at ./gcutils.jl:98 [inlined]
gc at ./gcutils.jl:98
unknown function (ip: 0x2ac301df91af)
_jl_invoke at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2758 [inlined]
ijl_apply_generic at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2940
jl_apply at /nopt/nrel/apps/julia/julia-1.9.3/src/julia.h:1880 [inlined]
do_call at /nopt/nrel/apps/julia/julia-1.9.3/src/interpreter.c:126
eval_value at /nopt/nrel/apps/julia/julia-1.9.3/src/interpreter.c:226
eval_stmt_value at /nopt/nrel/apps/julia/julia-1.9.3/src/interpreter.c:177 [inlined]
eval_body at /nopt/nrel/apps/julia/julia-1.9.3/src/interpreter.c:606
jl_interpret_toplevel_thunk at /nopt/nrel/apps/julia/julia-1.9.3/src/interpreter.c:762
jl_toplevel_eval_flex at /nopt/nrel/apps/julia/julia-1.9.3/src/toplevel.c:912
jl_toplevel_eval_flex at /nopt/nrel/apps/julia/julia-1.9.3/src/toplevel.c:856
ijl_toplevel_eval_in at /nopt/nrel/apps/julia/julia-1.9.3/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
eval_user_input at /nopt/nrel/apps/julia/julia-1.9.3/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:153
repl_backend_loop at /nopt/nrel/apps/julia/julia-1.9.3/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:249
#start_repl_backend#46 at /nopt/nrel/apps/julia/julia-1.9.3/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:234
start_repl_backend at /nopt/nrel/apps/julia/julia-1.9.3/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:231
_jl_invoke at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2758 [inlined]
ijl_apply_generic at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2940
#run_repl#59 at /nopt/nrel/apps/julia/julia-1.9.3/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:379
run_repl at /nopt/nrel/apps/julia/julia-1.9.3/usr/share/julia/stdlib/v1.9/REPL/src/REPL.jl:365
jfptr_run_repl_59762 at /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2758 [inlined]
ijl_apply_generic at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2940
#1017 at ./client.jl:421
jfptr_YY.1017_49406 at /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2758 [inlined]
ijl_apply_generic at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2940
jl_apply at /nopt/nrel/apps/julia/julia-1.9.3/src/julia.h:1880 [inlined]
jl_f__call_latest at /nopt/nrel/apps/julia/julia-1.9.3/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:819 [inlined]
invokelatest at ./essentials.jl:816 [inlined]
run_main_repl at ./client.jl:405
exec_options at ./client.jl:322
_start at ./client.jl:522
jfptr__start_33270 at /nopt/nrel/apps/julia/julia-1.9.3/usr/lib/julia/sys.so (unknown line)
_jl_invoke at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2758 [inlined]
ijl_apply_generic at /nopt/nrel/apps/julia/julia-1.9.3/src/gf.c:2940
jl_apply at /nopt/nrel/apps/julia/julia-1.9.3/src/julia.h:1880 [inlined]
true_main at /nopt/nrel/apps/julia/julia-1.9.3/src/jlapi.c:573
jl_repl_entrypoint at /nopt/nrel/apps/julia/julia-1.9.3/src/jlapi.c:717
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x400c08)
Allocations: 252297592 (Pool: 252265346; Big: 32246); GC: 96
Aborted (core dumped)
I have been able to reproduce this issue to a point if I manually call GC.gc() after the calculation of large systems but haven’t been able to do it for smaller cases where this doesn’t happen. The HPC node has 720 Gb of RAM and tracking the memory usage there seems no to be any problems with the memory availability.
Has anyone else run into this?