Corrupted double-linked list DifferentialEquations.jl

mm-crj · September 6, 2024, 11:51am

I am trying to solve a large system of odes as a finite volume discretisation of the heat equation on an unstructured mesh. However, as I approach the 10-15k equations mark, by deceasing grid size the solvers start to fail. It works fine for coarser meshes. I am posting the error log below, and I will try to come up with a minimal working example ASAP.

nohup: ignoring input
Info    : Reading 'meshes/rectangular[-5,5]unstruc/mesh5.msh'...
Info    : 8609 nodes
Info    : 17220 elements
Info    : Done reading 'meshes/rectangular[-5,5]unstruc/mesh5.msh'
ODE   0%|                                               |  ETA: N/A
ODE   0%|                                               |  ETA: 777.02 days
ODE   0%|                                               |  ETA: 736.09 days
ODE   0%|                                               |  ETA: 735.78 days
ODE   0%|                                               |  ETA: 722.17 days
ODE   0%|                                               |  ETA: 600.49 days
ODE   0%|                                               |  ETA: 428.67 days
ODE   0%|                                               |  ETA: 314.20 days
ODE   0%|                                               |  ETA: 245.55 days
ODE   0%|                                               |  ETA: 132.03 days
ODE   0%|                                               |  ETA: 83.19 days
ODE   0%|                                               |  ETA: 60.72 days
ODE   0%|                                               |  ETA: 47.82 days
ODE   0%|                                               |  ETA: 39.44 days
ODE   0%|                                               |  ETA: 30.07 days
ODE   0%|                                               |  ETA: 23.21 days
ODE   0%|                                               |  ETA: 18.91 days
ODE   0%|                                               |  ETA: 15.95 days
ODE   0%|                                               |  ETA: 13.73 days
ODE   0%|                                               |  ETA: 11.93 days
ODE   0%|                                               |  ETA: 10.07 days
ODE   0%|                                               |  ETA: 8 days, 15:12:59
ODE   0%|                                               |  ETA: 7 days, 13:20:33
ODE   0%|â                                              |  ETA: 6 days, 14:53:32
ODE   0%|â                                              |  ETA: 5 days, 21:22:51
ODE   0%|â                                              |  ETA: 5 days, 7:24:51
ODE   0%|â                                              |  ETA: 4 days, 18:50:26
ODE   0%|â                                              |  ETA: 4 days, 8:32:01
ODE   0%|â                                              |  ETA: 3 days, 22:45:27
ODE   0%|â                                              |  ETA: 3 days, 14:39:35
ODE   0%|â                                              |  ETA: 3 days, 7:57:57
ODE   0%|â                                              |  ETA: 3 days, 1:32:41
ODE   0%|â                                              |  ETA: 2 days, 19:32:03
ODE   0%|â                                              |  ETA: 2 days, 14:25:17
ODE   0%|â                                              |  ETA: 2 days, 10:03:04
ODE   0%|â                                              |  ETA: 2 days, 5:47:54
ODE   0%|â                                              |  ETA: 2 days, 1:44:40
ODE   0%|â                                              |  ETA: 1 days, 22:15:42
ODE   1%|â                                              |  ETA: 1 days, 19:12:45
ODE   1%|â                                              |  ETA: 1 days, 16:12:02
ODE   1%|â                                              |  ETA: 1 days, 13:14:38
ODE   1%|â                                              |  ETA: 1 days, 10:41:37
ODE   1%|â                                              |  ETA: 1 days, 8:26:55
ODE   1%|â                                              |  ETA: 1 days, 5:57:35
ODE   1%|â                                              |  ETA: 1 days, 3:49:45
ODE   1%|â                                              |  ETA: 1 days, 1:56:43
ODE   1%|â                                              |  ETA: 23:55:00
ODE   1%|â                                              |  ETA: 22:11:15
ODE   1%|â                                              |  ETA: 20:38:05
ODE   1%|â                                              |  ETA: 18:58:08
ODE   1%|â                                              |  ETA: 17:33:13
ODE   1%|â                                              |  ETA: 16:16:21
ODE   2%|â                                              |  ETA: 14:54:47
ODE   2%|â                                              |  ETA: 13:45:51
ODE   2%|â                                              |  ETA: 12:43:26
ODE   2%|â                                              |  ETA: 11:37:38
ODE   2%|â                                              |  ETA: 10:42:18
ODE   2%|ââ                                             |  ETA: 9:51:53
ODE   3%|ââ                                             |  ETA: 8:59:02
ODE   3%|ââ                                             |  ETA: 8:14:42
ODE   3%|ââ                                             |  ETA: 7:33:47
ODE   3%|ââ                                             |  ETA: 7:35:04
ODE   3%|ââ                                             |  ETA: 7:19:58
ODE   4%|ââ                                             |  ETA: 6:38:44
ODE   4%|ââ                                             |  ETA: 5:56:05
ODE   5%|âââ                                            |  ETA: 5:10:39
ODE   6%|âââ                                            |  ETA: 4:20:48
ODE   7%|ââââ                                           |  ETA: 3:21:47
ODE  11%|ââââââ                                         |  ETA: 2:04:42
ODE  21%|ââââââââââ                                     |  ETA: 0:59:41
ODE  46%|ââââââââââââââââââââââ                         |  ETA: 0:19:09
ODE 100%|âââââââââââââââââââââââââââââââââââââââââââââââ| Time: 0:16:22
ODE 100%|âââââââââââââââââââââââââââââââââââââââââââââââ| Time: 0:16:27
992.100210 seconds (11.43 G allocations: 284.965 GiB, 10.36% gc time, 4.96% compilation time)
error: <inline asm>:1:3: invalid character in input
         Q
         ^


[231233] signal (11.128): Segmentation fault
in expression starting at none:0
corrupted double-linked list

[231233] signal (6.-6): Aborted
in expression starting at none:0
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f5fef75926d)
unknown function (ip: 0x7f5fef7612fb)
unknown function (ip: 0x7f5fef76194b)
unknown function (ip: 0x7f5fef762e8a)
unknown function (ip: 0x7f5fef712952)
exit at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZN4llvm11LLVMContext8diagnoseERKNS_14DiagnosticInfoE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNSt17_Function_handlerIFvRKN4llvm12SMDiagnosticEbRKNS0_9SourceMgrERSt6vectorIPKNS0_6MDNodeESaISA_EEEZNS0_28MachineModuleInfoWrapperPass16doInitializationERNS0_6ModuleEEUlS3_bS6_SD_E_E9_M_invokeERKSt9_Any_dataS3_ObS6_SD_ at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm9MCContext8diagnoseERKNS_12SMDiagnosticE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser11DiagHandlerERKN4llvm12SMDiagnosticEPv at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageERNS_11raw_ostreamENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS8_INS_7SMFixItEEEb at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS6_INS_7SMFixItEEEb at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser3RunEbb at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmENS_9StringRefERKNS_15MCSubtargetInfoERKNS_15MCTargetOptionsEPKNS_6MDNodeENS_9InlineAsm10AsmDialectE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmEPKNS_12MachineInstrE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm10AsmPrinter16emitFunctionBodyEv at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /home/mainak/.local/julia-1.9.1/bin/../lib/julia/libLLVM-14jl.so (unknown line)

I am not sure if its a segfault as it ran out of memory. This is for alg = QNDF()/FBDF() at dt =1e-8 and abstol =1e-7, rtol = 1e-7 for about 16k equations.
Even a diagonisis of the problem would be a lot of help. Thanks!

ChrisRackauckas · September 6, 2024, 12:07pm

Try v1.10.5

mm-crj · September 6, 2024, 12:09pm

Okay, let me try that. Will post an update.

Update: It started to work after the update. Thanks a lot, I was just about to give up!

nohup: ignoring input
Info    : Reading 'meshes/rectangular[-5,5]unstruc/mesh5.msh'...
Info    : 8609 nodes
Info    : 17220 elements
Info    : Done reading 'meshes/rectangular[-5,5]unstruc/mesh5.msh'
ODE   0%|                                               |  ETA: N/A
ODE   0%|                                               |  ETA: 781960.09 days
ODE   0%|                                               |  ETA: 72041.40 days
ODE   0%|                                               |  ETA: 7228.95 days
ODE   0%|                                               |  ETA: 731.19 days
ODE   0%|                                               |  ETA: 74.03 days
ODE   0%|                                               |  ETA: 25.12 days
ODE   0%|                                               |  ETA: 15.08 days
ODE   0%|                                               |  ETA: 9 days, 0:49:15
ODE   0%|                                               |  ETA: 6 days, 10:44:28
ODE   0%|▏                                              |  ETA: 4 days, 1:14:48
ODE   0%|▏                                              |  ETA: 2 days, 22:25:16
ODE   0%|▏                                              |  ETA: 2 days, 7:10:32
ODE   1%|▎                                              |  ETA: 1 days, 8:29:24
ODE   1%|▍                                              |  ETA: 22:52:19
ODE   2%|▉                                              |  ETA: 10:18:35
ODE   5%|██▎                                            |  ETA: 4:13:56
ODE  12%|█████▌                                         |  ETA: 1:38:12
ODE  18%|████████▋                                      |  ETA: 0:57:03
ODE  32%|███████████████▏                               |  ETA: 0:27:25
ODE  46%|█████████████████████▊                         |  ETA: 0:15:14
ODE  73%|██████████████████████████████████▌            |  ETA: 0:04:49
ODE 100%|███████████████████████████████████████████████| Time: 0:13:15
ODE 100%|███████████████████████████████████████████████| Time: 0:13:16
ERROR: LoadError: There was an error during testing
in expression starting at /home/mainak/interacting-species-code/linear_diffusion/linear_diff_tri_V_work.jl:369
800.351042 seconds (25.76 G allocations: 660.256 GiB, 61.43% gc time, 5.58% compilation time)
Test Failed at /home/mainak/interacting-species-code/linear_diffusion/linear_diff_tri_V_work.jl:370
  Expression: minimum(sol.u[t]) >= 0.0
   Evaluated: -4.299619821511528e-14 >= 0.0

ChrisRackauckas · September 6, 2024, 6:03pm

One of the earlier Julia patches had this GC bug, so yeah it required an update. But it’s random if you hit it because it’s a GC thing, so it’s also kind of rare, so most things would work.

mm-crj · September 10, 2024, 9:13am

I am not so familiar with Julia Dev, so I was not aware about the GC thing. As per conventional wisdom, if things work, I usually don’t update them. So, thank you for pointing out the problem.