ModelingToolkit takes forever to precompile on Windows 11

Tooling Package Management
82

Weird and weirder… I’m trying to do some minimal work with Julia v10.3

using DataInterpolations
using Plots, LaTeXStrings, Latexify
using DataFrames, CSV

I have deleted the compiled and environments directories of .julia, and then used Pkg to add only the above packages – nothing else.

When I try to run the above code, I get the following error message:

ERROR: LoadError: ArgumentError: Package SpecialFunctions [276daf66-3868-5448-9aa4-cd146d93841b] is required but does not seem to be installed:
 - Run `Pkg.instantiate()` to install all recorded dependencies.

Stacktrace:
  [1] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:1926
  [2] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:1812
  [3] #invoke_in_world#3
    @ .\essentials.jl:926 [inlined]
  [4] invoke_in_world
    @ .\essentials.jl:923 [inlined]
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:1803
  [6] macro expansion
    @ .\loading.jl:1790 [inlined]
  [7] macro expansion
    @ .\lock.jl:267 [inlined]
  [8] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1753
  [9] #invoke_in_world#3
    @ .\essentials.jl:926 [inlined]
 [10] invoke_in_world
    @ .\essentials.jl:923 [inlined]
 [11] require(into::Module, mod::Symbol)
...
 [25] top-level scope
    @ stdin:3
in expression starting at C:\Users\Bernt\.julia\packages\DataInterpolations\2GasK\src\DataInterpolations.jl:1
in expression starting at stdin:3
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

There is a Project.toml and Manifest.toml file in the actual directory of the Jupyter notebook, but I want to use the global *.toml files – not the local ones!

This has worked in the past.

The above packages are not super large, so why is this error triggered on the above code, that used to run a week ago on Julia v10.3?

84

Why did you delete environments? Please be careful when deleting directories.

The global TOML files were in the environments directory. They are gone now.

Just do this.

85

Why are they gone? Yes, I deleted them [to make sure I have a clean install], but don’t they get re-created when I add the packages again and the packages get precompiled?? I added them before I ran the code that lead to “instantiation” request.

86

Apparently you forgot something or you deleted the packages directory as well? Anyways, just run Pkg.instantiate(). I’m really not sure why you do not want to do that. It will make sure that whatever you have in your TOML files is actually installed.

87

I don’t mind doing Pkg.instantiate, but I need to know what the consequence is. My intention was to use the global environment, and not the local environment *.toml files [which uses Julia v.9.x, or something]. So if I had done Pkg.instantiate, that would have installed the packages listed in the local *.toml files, and that is not what I want.

88

The effect is that all packages and their dependencies from Package.toml get installed if they are not yet installed.

89

The discussion seems to be spread over two threads. I first posted it here, but then deleted it and moved it to another thread. But the answers seems to pop up here :slight_smile:

So let me cross-post my latest in the other thread (since most answers are here)…

OK – Pkg.status() tells me that I’m in the local directory!!

C:\Users\Bernt\OneDrive\...\Notebooks\Project.toml` (empty project)

This is so, even though I did not do Pkg.activate(".")!!

  • So this implies that (at least when I run Jupyter Notebooks in VSCode), command Pkg.activate(".") is not needed – it will always first search for the *.toml files locally!
  • If there is no *.toml files locally, it will go to the global environment and use the *.toml files there.

This also implies that I can not use my strategy of having multiple related notebooks in one directory where one of them uses a local environment! (Unless I explicitly specify the global environment in every notebook that do not use the local one?)

Perhaps I misunderstand the role of Pkg.activate()… I thought that was meant to specify what is the active environment, and if unspecified, the global environment is used. But instead, it seems to just be to activate the creation of local *.toml files??

90

So – I understand the role of Pkg.instantiate [I think], but I need to know which Project.toml and Manifest.toml file is used.

It seems like when I run Jupyter notebooks, then Julia defaults to looking in the directory where the notebook is located, and not in the location of the global Project.toml and Manifest.toml files [which I thought it would].

I thought that if I had not used Pkg.activate("...") to specify a location for the Project.toml and Manifest.toml files, then the global files would be used. But it seems to not be so.

91

Let’s discuss this Pkg usage in the other thread, please.

92

As an update, there was an issue in the linker that was causing this. Set storage class of julia globals to dllimport on windows to avoid auto-import weirdness by gbaraldi · Pull Request #54572 · JuliaLang/julia · GitHub fixes the issue, and there will be a hotfix in Julia which which should follow as a Julia v1.10.4 ASAP.

Note that this is a Windows-only bug.

The linker bug is triggered seemingly based on the size of the image being compiled. There were a few changes that might’ve put us into this regime, in particular the new ODE defaulting system Redesign default ODE solver to be type-grounded and lazy by oscardssmith · Pull Request #2184 · SciML/OrdinaryDiffEq.jl · GitHub. For these changes, we’ve disabled them on Windows in the latest releases Disable Default solver on windows by oscardssmith · Pull Request #2224 · SciML/OrdinaryDiffEq.jl · GitHub. There isn’t all that much we can do to workaround the bug beyond this, but this should hopefully make it trigger for less people less often.

6 Likes
93

fixed for real in Julia 1.10.4

10 Likes
94

All my listed packages above now precompile. I’ll test them, but first an ice cream excursion in the beautiful weather…

95

… and: confirming that ModelingToolkit works.

1 Like
96

Thanks everyone for helping us isolate this and coming together as a community, also thanks for the patience in getting the fix out there. This was a really rough one but given how smooth relatively it ended up going. This thread to me is a shining point of our community for how the Julia compiler team, the SciML crew, and community help all came together in what was both very scary but also very solvable!

15 Likes
97

And… thanks to Chris and others for “fumbling in the dark” initially but quickly focusing in on where the problem lay + the Julia developers who quickly responded with a fix.

I missed running ModelingToolkit and other tools for some days. Fortunately, there were other packages that still worked under Julia v10.3 so I had things to keep me busy.

To me, this shows how dependent I have become of Julia.

4 Likes