ModelingToolkit takes forever to precompile on Windows 11

ufechner7 · May 23, 2024, 2:37am

Did you create a bug report?

mkitti · May 23, 2024, 3:25am

Do you know if it is OrdinaryDiffEq for sure or is it that OrdinaryDiffEq now depends on?

ChrisRackauckas · May 23, 2024, 3:29am

Just an update to everyone. I’m traveling but I’m keeping investigations of this going. It is definitely a weird one. Root cause seems to be a very obscure GC bug when mixed with an edge case of metaprogramming. Quoting @gbaraldi:

So the GC bug we are looking at is really bizarre. Somewhere we have a quote node like quote nothing end but the value in the quote node isn’t a pointer to the nothing value, but the address of jl_nothing (the C value) is whats there, and we get super lucky that jl_nothing_type is stored in the binary 8 bytes before so a jl_astagged_value(&jl_nothing) gets jl_nothing_type , so it doesn’t crash instantly. In fact the GC tries to mark that quote node and sees a value with a valid type, but instead of that being a type tag, it’s the C address

So it sounds like a more general solution is coming to Julia, but we also of course don’t want people to be waiting on this.

However, tracking down what exactly is doing this is still a bit elusive. @Brad_Carman bound that rolling back OrdinaryDiffEq to v6.75.0 fixes the problem. What that’s the case? Who knows right now, but I started a CI-based bisection:

I am not home so I don’t have a Windows computer on me but this will let us know if there’s a specific commit involved.

ChrisRackauckas · May 23, 2024, 4:02am

CI reports show no issue with precompilation of either version on latest Windows…

Datseris · May 23, 2024, 10:24am

Thanks for looking at it. I can confrim that pinning OrdinaryDiffEq.jl to v6.75 makes MTK compile. Wow, so odd.

ChrisRackauckas · May 23, 2024, 11:14am

Can someone who has this issue please bisect to the commit that is causing it? Since it doesn’t show up on CI and I’m traveling without a windows computer right now I’m blind.

anguille · May 23, 2024, 11:51am

Hi,
I have ModelingToolkit precompiling fine with OrdinaryDiffEq 6.75.0 and 6.76.0.
Upgrading to 6.77.0 causes issues:

Then shutting down julia and running a test script with “using ModelingToolkit”:

Precompiling ModelingToolkit
        Info Given ModelingToolkit was explicitly requested, output will be shown live

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffb847676bf -- gc_setmark_pool_ at C:/workdir/src\gc.c:876 [inlined]
gc_setmark_pool at C:/workdir/src\gc.c:895 [inlined]
gc_setmark at C:/workdir/src\gc.c:902 [inlined]
gc_setmark at C:/workdir/src\gc.c:898 [inlined]
gc_mark_outrefs at C:/workdir/src\gc.c:2606 [inlined]
gc_mark_loop_serial_ at C:/workdir/src\gc.c:2679
in expression starting at C:\Users\guill\.julia\packages\ModelingToolkit\BAG22\src\ModelingToolkit.jl:6
gc_setmark_pool_ at C:/workdir/src\gc.c:876 [inlined]
gc_setmark_pool at C:/workdir/src\gc.c:895 [inlined]
gc_setmark at C:/workdir/src\gc.c:902 [inlined]
gc_setmark at C:/workdir/src\gc.c:898 [inlined]
gc_mark_outrefs at C:/workdir/src\gc.c:2606 [inlined]
gc_mark_loop_serial_ at C:/workdir/src\gc.c:2679
gc_mark_loop_serial at C:/workdir/src\gc.c:2702
gc_mark_loop at C:/workdir/src\gc.c:2897 [inlined]
_jl_gc_collect at C:/workdir/src\gc.c:3221
ijl_gc_collect at C:/workdir/src\gc.c:3518
maybe_collect at C:/workdir/src\gc.c:937 [inlined]
jl_gc_pool_alloc_inner at C:/workdir/src\gc.c:1293
jl_gc_pool_alloc_noinline at C:/workdir/src\gc.c:1350 [inlined]
jl_gc_alloc_ at C:/workdir/src\julia_internal.h:477 [inlined]
jl_gc_alloc at C:/workdir/src\gc.c:3570
jl_gc_alloc_buf at C:/workdir/src\julia_internal.h:515 [inlined]
jl_gc_alloc_buf at C:/workdir/src\julia_internal.h:513 [inlined]
array_resize_buffer at C:/workdir/src\array.c:698
jl_array_grow_at_end at C:/workdir/src\array.c:893
_growend! at .\array.jl:1072 [inlined]
push! at .\array.jl:1119 [inlined]
construct_ssa! at .\compiler/ssair\slot2ssa.jl:691
slot2reg at .\compiler\optimize.jl:626
run_passes at .\compiler\optimize.jl:489
run_passes at .\compiler\optimize.jl:504 [inlined]
optimize at .\compiler\optimize.jl:453 [inlined]
_typeinf at .\compiler\typeinfer.jl:272
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_unknown at .\compiler\abstractinterpretation.jl:2152
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_unknown at .\compiler\abstractinterpretation.jl:2152
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_unknown at .\compiler\abstractinterpretation.jl:2152
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_unknown at .\compiler\abstractinterpretation.jl:2152
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_unknown at .\compiler\abstractinterpretation.jl:2152
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_unknown at .\compiler\abstractinterpretation.jl:2152
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_edge at .\compiler\typeinfer.jl:930
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_known at .\compiler\abstractinterpretation.jl:2083
abstract_call at .\compiler\abstractinterpretation.jl:2165
abstract_call at .\compiler\abstractinterpretation.jl:2158
abstract_call at .\compiler\abstractinterpretation.jl:2350
abstract_eval_call at .\compiler\abstractinterpretation.jl:2366
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2376
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2620
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2909
typeinf_local at .\compiler\abstractinterpretation.jl:3094
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3182
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_ext at .\compiler\typeinfer.jl:1051
typeinf_ext_toplevel at .\compiler\typeinfer.jl:1082
typeinf_ext_toplevel at .\compiler\typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_39162.1 at C:\Users\guill\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
_jl_invoke at C:/workdir/src\gf.c:2894 [inlined]
ijl_apply_generic at C:/workdir/src\gf.c:3076 [inlined]
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
jl_type_infer at C:/workdir/src\gf.c:394
_generate_from_hint at C:/workdir/src\gf.c:2755 [inlined]
jl_compile_now at C:/workdir/src\gf.c:2769 [inlined]
ijl_compile_method_instance at C:/workdir/src\gf.c:2781
ijl_compile_hint at C:/workdir/src\gf.c:2819
precompile at .\loading.jl:3204
precompile_mi at C:\Users\guill\.julia\packages\PrecompileTools\L8A3n\src\PrecompileTools.jl:13
foreach at .\abstractarray.jl:3094
recompile_invalidations at C:\Users\guill\.julia\packages\PrecompileTools\L8A3n\src\invalidations.jl:23
unknown function (ip: 000002b798781973)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_call at C:/workdir/src\interpreter.c:126
eval_value at C:/workdir/src\interpreter.c:223
eval_stmt_value at C:/workdir/src\interpreter.c:174 [inlined]
eval_body at C:/workdir/src\interpreter.c:635
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:775
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:934
jl_eval_module_expr at C:/workdir/src\toplevel.c:215 [inlined]
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:736
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
ijl_toplevel_eval at C:/workdir/src\toplevel.c:943 [inlined]
eval_body at C:/workdir/src\interpreter.c:635
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:775
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:934
jl_eval_module_expr at C:/workdir/src\toplevel.c:215 [inlined]
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:736
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
ijl_toplevel_eval at C:/workdir/src\toplevel.c:943 [inlined]
eval_body at C:/workdir/src\interpreter.c:635
eval_body at C:/workdir/src\interpreter.c:635
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:775
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:934
jl_eval_module_expr at C:/workdir/src\toplevel.c:215 [inlined]
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:736
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
ijl_toplevel_eval at C:/workdir/src\toplevel.c:943 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:985
eval at .\boot.jl:385 [inlined]
include_string at .\loading.jl:2070
_include at .\loading.jl:2130
include at .\Base.jl:495 [inlined]
include_package_for_output at .\loading.jl:2216
jfptr_include_package_for_output_84978.1 at C:\Users\guill\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_call at C:/workdir/src\interpreter.c:126
eval_value at C:/workdir/src\interpreter.c:223
eval_stmt_value at C:/workdir/src\interpreter.c:174 [inlined]
eval_body at C:/workdir/src\interpreter.c:635
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:775
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:934
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
ijl_toplevel_eval at C:/workdir/src\toplevel.c:943 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:985
eval at .\boot.jl:385 [inlined]
include_string at .\loading.jl:2070
include_string at .\loading.jl:2080 [inlined]
exec_options at .\client.jl:316
_start at .\client.jl:552
jfptr__start_86719.1 at C:\Users\guill\.julia\juliaup\julia-1.10.0+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
true_main at C:/workdir/src\jlapi.c:582
jl_repl_entrypoint at C:/workdir/src\jlapi.c:731
mainCRTStartup at C:/workdir/cli\loader_exe.c:58
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 14434620 (Pool: 14417009; Big: 17611); GC: 18
^CProgress [>                                        ]  0/1
  ◐ ModelingToolkit

and hangs for more than 20min.
If I stop the process and start again my script works ‘immediatly’.

BLI · May 23, 2024, 11:55am

I’d do it if I had a clue as to what bisecting on GitHub means (I’m a simplistic GitHub user…). Anyone with some experience? Only realistic way I could contribute is to set up a Teams meeting where I share my screen and have someone tell me what to do, which may not be efficient…

ChrisRackauckas · May 23, 2024, 12:08pm

Okay, so now there’s conflicting reports as to what version it was. Fantastic. @Brad_Carman are you absolutely positive v6.76 failed for you and you needed exactly v6.75?

Bisect means going commit-by-commit and trying it out. So dev the OrdinaryDiffEq repo and, if the latest report is correct, start at 6.76 and confirm it works, then do 6.77 and confirm it crashes. Then take some commit in the middle: does it work or give the crash? If it crashes, you know the first half of the commit has the commit. Keep doing that and it’ll point to the exact commit that does the code change that causes the crash. That will then point us to exactly what is causing the problem.

ChrisRackauckas · May 23, 2024, 12:16pm

The changes from v6.76 to v6.77 includes 3 PRs:

github.com/SciML/OrdinaryDiffEq.jl

shorten timespan on initdt test

SciML:master ← oscardssmith:patch-3

opened 02:21PM - 16 May 24 UTC

oscardssmith

+1 -1

This previously was testing that you don't get a warning even though the solutio…n was `Inf`. The only reason it worked at all is that `Euler` is simple enough that it stayed `Inf` rather than going to `NaN` and DiffEqBase incorrectly didn't consider `Inf` to be diverged. ## Checklist - [ ] Appropriate tests were added - [ ] Any code changes were done in a way that does not break public API - [ ] All documentation related to code changes were updated - [ ] The new code follows the [contributor guidelines](https://github.com/SciML/.github/blob/master/CONTRIBUTING.md), in particular the [SciML Style Guide](https://github.com/SciML/SciMLStyle) and [COLPRAC](https://github.com/SciML/COLPRAC). - [ ] Any new documentation only uses public API ## Additional context Add any other context about the problem here.

The last one only changes a test value, so it can safely be ignored.

The second one doesn’t touch anything with a macro so I assume there’s nothing that can trigger the GC issue @gbaraldi mentioned, so I think that one is off the list. But this is exactly what a bisection can confirm.

So given the latest post there, there’s only one PR that it could be, and since it’s 3 commits with 2 commits just being clean up (formatting), my guess is that it’s this commit:

That might be pointing that you can hit this very obscure GC bug when you mix @views @.. macros?

Now to confirm this, @anguille or someone else who is hitting this, can you please ]dev OrdinaryDiffEq and use git to check out this PR branch:

When you run with the latest version of the package with that PR branch, does the crash go away?

Datseris · May 23, 2024, 12:20pm

If noone has done it tonight, I will do it! (but now I really have to finish some work)

BLI · May 23, 2024, 12:25pm

Ok: will try when I’m back with laptop after lunch… ca within 1/2 hour.

I assume I first add OrdinaryDiffEq of a certain version prior to adding ModelingToolkit, i.e. that this is what “pin” means.

sjstreicher · May 23, 2024, 12:30pm

I had this issue on Julia 1.10.3 ~~(even after pinning OrdinaryDiffEq@6.75)~~, but was able to compile ModelingToolkit successfully on Julia 1.11.0-beta1 without pinning any packages.

Update: Apologies, I added the specific version, but did not use pin, works for 6.75.

Brad_Carman · May 23, 2024, 12:34pm

Just want to add that what I know for sure is that v6.75.0 worked for me, and v6.77.0 didn’t. I have no certainty about v6.76.0. I will try to confirm that this version works when I get a chance.

anguille · May 23, 2024, 12:35pm

ChrisRackauckas · May 23, 2024, 12:42pm

Go one commit forward. That’s the cleanup commit I mentioned. Try this commit:

Brad_Carman · May 23, 2024, 12:49pm

I tried with a clean environment

(MTKCrash) pkg> add https://github.com/SciML/OrdinaryDiffEq.jl.git#crash_fix
(MTKCrash) pkg> add ModelingToolkit

Doesn’t seem to be working, MTK has been precompiling for about 15 minutes now.

anguille · May 23, 2024, 12:51pm

All good with this commit:

I can also confirm that the one before this branch is OK:

Will try the next one (3rd commit of the dprkn6_interp):

ChrisRackauckas · May 23, 2024, 1:03pm

Don’t tell me the formatter found an obscure GC bug that’s only on Windows when macros hit a specific edge case.

Brad_Carman · May 23, 2024, 1:03pm

Yes, I can confirm that those 3 commits are all good on my end.

(MTKCrash) pkg> add https://github.com/SciML/OrdinaryDiffEq.jl.git#2e7cb8c89ab9b0f0b013dc04d5ce6f0e01dd5148
   Resolving package versions...
    Updating `C:\Work\Temp\MTKCrash\Project.toml`
  [1dea7af3] ~ OrdinaryDiffEq v6.78.0 `https://github.com/SciML/OrdinaryDiffEq.jl.git#crash_fix` ⇒ v6.76.0 `https://github.com/SciML/OrdinaryDiffEq.jl.git#2e7cb8c`
    Updating `C:\Work\Temp\MTKCrash\Manifest.toml`
  [1dea7af3] ~ OrdinaryDiffEq v6.78.0 `https://github.com/SciML/OrdinaryDiffEq.jl.git#crash_fix` ⇒ v6.76.0 `https://github.com/SciML/OrdinaryDiffEq.jl.git#2e7cb8c`
Precompiling project...
  2 dependencies successfully precompiled in 226 seconds. 229 already precompiled.

So in summary:

branch crash_fix does not work
dprkn6_interp is OK