Sudden Falure to Precompile DifferentialEquations and Sundials

Tooling Package Management
1

Hi, after running well for months, my Julia programs suddenly cannot precompile DifferentialEquations or Sundials anymore, which is a fatal problem for my programs. I don’t know what triggered Julia to try (and fail) to Precompile all of my packages over again, except maybe because I just added & tried using the packages Gadfly and PGFPlotsX.

I get the error, LoadError: UndefVarError: initialize_dae! not defined, and searching online found an old thread about needing DiffEqBase greater than or equal to v6.21.0. When I enter the command, pkgs = Pkg.installed(); pkgs["DiffEqBase"], I get the result, v"6.19.0". But nothing I’ve tried to update DiffEqBase has worked to fix the problems, and just creates more error messages. (I will include a full error message stack trace in a follow-up message.)

I am working on a PC computer system running Microsoft Windows 10 Enterprise, Version 10.0.19042 Build 19042. But I’m getting the same error messages on an older computer system also, where I also (unfortunately) added the packages Gadfly and PGFPlotsX.

I would very much appreciate help to fix this situation, I cannot advance my work without fixing DifferentialEquations and Sundials. Thank you!

2

Hi again, here is a full stack trace of the error messages I get when I try to Precompile DifferentialEquations again now. If this makes sense to anyone, please let me know!

First, just the highlights:

[ Info: Precompiling DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
ERROR: LoadError: LoadError: UndefVarError: initialize_dae! not defined

ERROR: LoadError: Failed to precompile Sundials [c3572dad-4567-51f8-b174-8c6c989267f4] to C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\compiled\v1.4\Sundials\j8Ppj_GjXEj.ji.

[ Info: Precompiling Sundials [c3572dad-4567-51f8-b174-8c6c989267f4]
ERROR: LoadError: LoadError: UndefVarError: initialize_dae! not defined

Now, the full stack trace:

[ Info: Precompiling DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
ERROR: LoadError: LoadError: UndefVarError: initialize_dae! not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at .\Base.jl:26
 [2] top-level scope at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\common_interface\integrator_utils.jl:151
 [3] include(::Module, ::String) at .\Base.jl:377
 [4] include(::String) at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\Sundials.jl:3
 [5] top-level scope at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\Sundials.jl:71
 [6] include(::Module, ::String) at .\Base.jl:377
 [7] top-level scope at none:2
 [8] eval at .\boot.jl:331 [inlined]
 [9] eval(::Expr) at .\client.jl:449
 [10] top-level scope at .\none:3
in expression starting at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\common_interface\integrator_utils.jl:151
in expression starting at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\Sundials.jl:71
ERROR: LoadError: Failed to precompile Sundials [c3572dad-4567-51f8-b174-8c6c989267f4] to C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\compiled\v1.4\Sundials\j8Ppj_GjXEj.ji.
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at .\loading.jl:1272
 [3] _require(::Base.PkgId) at .\loading.jl:1029
 [4] require(::Base.PkgId) at .\loading.jl:927
 [5] require(::Module, ::Symbol) at .\loading.jl:922
 [6] include(::Module, ::String) at .\Base.jl:377
 [7] top-level scope at none:2
 [8] eval at .\boot.jl:331 [inlined]
 [9] eval(::Expr) at .\client.jl:449
 [10] top-level scope at .\none:3
in expression starting at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\DifferentialEquations\HSWeG\src\DifferentialEquations.jl:13
[ Info: Precompiling Sundials [c3572dad-4567-51f8-b174-8c6c989267f4]
ERROR: LoadError: LoadError: UndefVarError: initialize_dae! not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at .\Base.jl:26
 [2] top-level scope at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\common_interface\integrator_utils.jl:151
 [3] include(::Module, ::String) at .\Base.jl:377
 [4] include(::String) at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\Sundials.jl:3
 [5] top-level scope at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\Sundials.jl:71
 [6] include(::Module, ::String) at .\Base.jl:377
 [7] top-level scope at none:2
 [8] eval at .\boot.jl:331 [inlined]
 [9] eval(::Expr) at .\client.jl:449
 [10] top-level scope at .\none:3
in expression starting at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\common_interface\integrator_utils.jl:151
in expression starting at C:\Users\phybdb\.juliapro\JuliaPro_v1.4.2-1\packages\Sundials\0yJ3G\src\Sundials.jl:71
3

It seems you have already identified that the problem lies with DiffEqBase 6.19, so what happens when you ] add DiffEqBase@6.21?

1 Like
4

Hi nilshg, I tried that; I get massive errors:

(@v1.4) pkg> add DiffEqBase@6.21
   Updating registry at `C:\Users\.juliapro\JuliaPro_v1.4.2-1\registries\JuliaPro`
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package ProgressMeter [92933f4c]:
 ProgressMeter [92933f4c] log:
 ├─possible versions are: [0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0-1.3.4, 1.4.0-1.4.1, 1.5.0] or uninstalled
 ├─restricted by compatibility requirements with Mads [d6bdc55b] to versions: [0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0-1.3.4, 1.4.0-1.4.1, 1.5.0]
 │ └─Mads [d6bdc55b] log:
 │   ├─possible versions are: [0.5.2, 0.6.0-0.6.3, 0.7.0-0.7.6, 0.8.0, 0.9.0-0.9.7, 0.10.0-0.10.8, 1.0.0-1.0.10] or uninstalled
 │   ├─restricted to versions * by an explicit requirement, leaving only versions [0.5.2, 0.6.0-0.6.3, 0.7.0-0.7.6, 0.8.0, 0.9.0-0.9.7, 0.10.0-0.10.8, 1.0.0-1.0.10]  
 │   └─restricted by julia compatibility requirements to versions: [0.5.2, 0.6.0-0.6.3, 0.7.0-0.7.6, 0.8.0, 0.9.0-0.9.7, 0.10.0-0.10.8] or uninstalled, leaving only versions: [0.5.2, 0.6.0-0.6.3, 0.7.0-0.7.6, 0.8.0, 0.9.0-0.9.7, 0.10.0-0.10.8]
 ├─restricted by compatibility requirements with Mads [d6bdc55b] to versions: [0.6.0-0.6.1, 0.7.0, 0.8.0, 0.9.0]
 │ └─Mads [d6bdc55b] log: see above
 └─restricted by compatibility requirements with ConsoleProgressMonitor [88cd18e8] to versions: [1.0.0, 1.1.0, 1.2.0, 1.3.0-1.3.4, 1.4.0-1.4.1, 1.5.0] — no versions left
   └─ConsoleProgressMonitor [88cd18e8] log:
     ├─possible versions are: 0.1.0-0.1.2 or uninstalled
     ├─restricted by compatibility requirements with DiffEqBase [2b5f629d] to versions: 0.1.0-0.1.2
     │ └─DiffEqBase [2b5f629d] log:
     │   ├─possible versions are: [3.13.2-3.13.3, 4.0.0-4.0.1, 4.1.0, 4.2.0, 4.3.0-4.3.1, 4.4.0, 4.5.0, 4.6.0, 4.7.0, 4.8.0, 4.9.0, 4.10.0-4.10.1, 4.11.0-4.11.1, 4.12.0, 4.13.0, 4.14.0-4.14.1, 4.15.0, 4.16.0, 4.17.0, 4.18.0, 4.19.0, 4.20.0-4.20.3, 4.21.0, 4.21.2-4.21.3, 4.22.0-4.22.2, 4.23.0, 4.23.2-4.23.4, 4.24.0-4.24.3, 4.25.0-4.25.1, 4.26.0-4.26.3, 4.27.0-4.27.1, 4.28.0-4.28.1, 
4.29.0-4.29.2, 4.30.0-4.30.2, 4.31.0-4.31.2, 4.32.0, 5.0.0-5.0.1, 5.1.0, 5.2.0-5.2.3, 5.3.0-5.3.2, 5.4.0-5.4.1, 5.5.0-5.5.2, 5.6.0-5.6.4, 5.7.0, 5.8.0-5.8.1, 5.9.0, 5.10.0-5.10.3, 5.11.0-5.11.1, 5.12.0, 5.13.0, 5.14.0-5.14.2, 5.15.0, 5.16.0-5.16.5, 5.17.0-5.17.1, 5.18.0, 5.19.0, 5.20.0-5.20.1, 6.0.0, 6.1.0, 6.2.0-6.2.4, 6.3.0-6.3.6, 6.4.0-6.4.2, 6.5.0-6.5.1, 6.6.0, 6.7.0, 6.8.0, 6.9.0-6.9.4, 6.10.0-6.10.2, 6.11.0, 6.12.0-6.12.5, 6.13.0-6.13.3, 6.14.0-6.14.2, 6.15.0-6.15.2, 6.16.0, 6.17.0-6.17.3, 6.18.0-6.18.1, 6.19.0, 6.20.0, 6.21.0-6.21.1, 6.22.0-6.22.2, 6.23.0, 6.24.0, 6.25.0-6.25.2, 6.26.0, 6.27.0, 6.28.0, 6.29.0-6.29.3, 6.30.0-6.30.4, 6.31.0-6.31.1, 6.32.0-6.32.2, 6.33.0-6.33.1, 6.34.0-6.34.3, 6.35.0-6.35.2, 6.36.0-6.36.4, 6.37.0, 6.38.0-6.38.4, 6.39.0-6.39.1, 6.40.0-6.40.9, 6.41.0-6.41.3, 6.42.0, 6.43.0-6.43.1, 6.44.0-6.44.3, 6.45.0-6.45.1, 6.46.0-6.46.1, 6.47.0-6.47.1, 6.48.0-6.48.2, 6.49.0-6.49.2, 6.50.0, 6.51.0, 6.52.0-6.52.2, 6.53.0-6.53.6, 6.54.0-6.54.1, 6.55.0, 6.56.0, 6.57.0-6.57.9, 6.58.0, 6.59.0, 6.60.0] or uninstalled
     │   └─restricted to versions 6.21 by an explicit requirement, leaving only versions 6.21.0-6.21.1
     └─restricted by compatibility requirements with LoggingExtras [e6f89c97] to versions: 0.1.1-0.1.2 or uninstalled, leaving only versions: 0.1.1-0.1.2
       └─LoggingExtras [e6f89c97] log:
         ├─possible versions are: [0.1.0, 0.3.0, 0.4.0-0.4.6] or uninstalled
         └─restricted by compatibility requirements with DiffEqBase [2b5f629d] to versions: 0.4.0-0.4.6
           └─DiffEqBase [2b5f629d] log: see above

(@v1.4) pkg>
5

Perhaps I should also add the following status information:

julia> Pkg.status()
Status `C:\Users\JuliaPro_v1.4.2-1\environments\v1.4\Project.toml`
  [28f2ccd6] ApproxFun v0.11.14
  [7e558dbc] ArbNumerics v1.2.5
  [c52e3926] Atom v0.12.11 ⚲
  [7f9c7709] BIGUQ v0.8.0
  [fbb218c0] BSON v0.3.3
  [093aae92] BSplineKit v0.4.0
  [aae01518] BandedMatrices v0.15.15
  [a2e0e22d] CalculusWithJulia v0.0.1
  [8f4d0f93] Conda v1.5.2
  [2b5f629d] DiffEqBase v6.19.0
  [9fdde737] DiffEqOperators v4.12.0
  [0c46a032] DifferentialEquations v6.16.0
  [fdbdab4c] ElasticArrays v1.2.6
  [6a86dc24] FiniteDiff v2.8.0
  [26cc04aa] FiniteDifferences v0.10.4
  [f6369f11] ForwardDiff v0.10.18
  [28b8d3ca] GR v0.48.0
  [c91e804a] Gadfly v1.2.1
  [dcc97b0b] GeoStats v0.21.1
  [7073ff75] IJulia v1.23.2
  [a98d9a8b] Interpolations v0.12.10
  [d1acc4aa] IntervalArithmetic v0.16.7
  [9da8a3cd] JLSO v2.5.0
  [e5e0dc1b] Juno v0.8.2 ⚲
  [d293930c] KrigingEstimators v0.5.1
  [7f56f5a3] LSODA v0.6.1
  [d6bdc55b] Mads v0.10.8
  [961ee093] ModelingToolkit v2.0.0
  [2edaba10] Nemo v0.22.1
  [54ca160b] ODEInterface v0.4.8
  [09606e27] ODEInterfaceDiffEq v3.6.0
  [429524aa] Optim v0.20.1
  [1dea7af3] OrdinaryDiffEq v5.29.0
  [8314cec4] PGFPlotsX v1.2.10
  [4722fa14] PkgAuthentication v1.1.1
  [91a5bcdd] Plots v0.29.9
  [097a2a50] PolyFit v0.1.0 #master (https://github.com/jishnub/PolyFit.jl)
  [f27b6e38] Polynomials v0.6.1
  [438e738f] PyCall v1.92.3
  [1fd47b50] QuadGK v2.4.1
  [b0e4dd01] RollingFunctions v0.6.2
  [f2b01f46] Roots v1.0.9
  [47a9eef4] SparseDiffTools v1.13.2
  [684fba80] SparsityDetection v0.3.4
  [276daf66] SpecialFunctions v0.10.3
  [c3572dad] Sundials v4.1.1
  [92b13dbe] TaylorIntegration v0.8.8
  [04a0146e] Variography v0.10.4
  [44d3d7a6] Weave v0.10.7
  [ade2ca70] Dates
  [8bb1440f] DelimitedFiles
  [37e2e46d] LinearAlgebra
  [2f01184e] SparseArrays
6

Try this. Create a new environment with minimal Project.toml and Manifest.toml (I usually delete everything except the part that has your package info/uuid), reinstall the packages you need for this project, and only those.

One possible reason for the problems is that adding a package has downgraded another (something like this has happened to me a couple of times). Maybe you can get back to the way things were by removing the package and doing a Pkg.update(), but if that doesn’t work, the quickest way is simply to create a new environment. It doesn’t take that long (still, a few minutes) to reinstall the packages, because most of them will not need to be downloaded again, they are already in your .julia directory.

1 Like
7

In addition to the sound advice above, there should also be no reason to be using Julia version 1.4 - the current stable release is 1.6.1, so it’s probably worth updating.

1 Like
8

Hi ptoche, thanks for the suggestion. Creating an all-new environment seems like a bit of a nuclear option. Anyway, I did a little more research, particular into the prior topic:

https://discourse.julialang.org/t/a-guide-how-to-handle-error-unsatisfiable-requirements-detected/43406

And that explanation showed me how to properly interpret my stack trace above for add DiffEqBase@6.21. Looking through that shows that the package Mads seems to be the problem; so I did a Pkg.rm("Mads"), and then was able to update DiffEqBase, and I was able to bring back DifferentialEquations and Sundials.

Unfortunately, I then got the error, Could not load library "libGR.dll", which then had to be fixed with ENV["GRDIR"]="", and Pkg.build("GR").

Everything seems to work again now, so far, but I am holding my breath.

Thanks also for your advice, nilshg; I will update to Julia 1.6.1 as soon as necessary; for now, I’m afraid to change anything else unless I have to.

1 Like
9

The issues is Mads.jl’s CompatHelper was only looking for new packages that support Julia v1.3, so its Project.toml doesn’t include any recent packages.

https://github.com/madsjulia/Mads.jl/pull/43

2 Likes
10

1.6.1 is a minor release, and as such should not break anything you’re doing. On the other hand it can happen that packages drop compatibility to older Julia versions (as a newer version introduced some feature that the package author wants to leverage), which then restricts you to an older version of the package which is compatible with the outdated Julia version you’re using.

I’d say give it a try - the balance of risk and reward is in my opinion quite heavily skewed to the reward side here, plut you’re not really losing anything as it probably only takes half an hour to install Julia 1.6.1 alongside your current version and test drive your code in it!

Also apart from that, I would not refer to a new environment as a “nuclear option”, on the contrary it should be your preferred way of working in Julia. Your posts above suggest that you are working in the default environment (@v1.4), which is not recommended. It massively increases the risk of the kind of dependency conflicts you are experiencing, as you are forcing all packages you’re using (even if you never use them together on the same piece of code) to be compatible with another. It also makes it harder to keep your work reproducible - if you write some code which depends on packages in your default environment, it’s very hard to get it to run again a year later when your default environment has moved on. If instead you’d use an environment, then the code would use more or less exactly (depending on whether you rely on the Project.toml or Manifest.toml) the versions of all packages that you used whenever you last updated packages in that specific environment.

I started a recent thread about this here: Should we have a package update PSA? as I think this is an underappreciated aspect of the Julia workflow for many people.

1 Like
11

All true. Package management is a subtle science and sooner or later a package incompatibility will crop up and cause a situation similar to the one you’ve just experienced. To minimize this risk, you should create an environment per project, you could have a “trash” environment in which you play around with packages for the first time or even use temporary environments. See the docs for more info.

Setting up environments is not complicated and in the long run it will make your life easier. By keeping a record of the package versions in the Project.toml and Manifest.toml, it guarantees that you will be able to run your code again in a couple of years time without having to edit (with the appropriate version of Julia of course). In each of my projects I start with something like:

dir = expanduser("~/Julia/Experiments/")
push!(LOAD_PATH, dir)
import Pkg
Pkg.activate(dir)
Base.active_project()

This will print the active environment to remind me where I am before I do anything like Pkg.add("this") or Pkg.update().

You can add strict compat bounds in your Project.toml to minimize the risk of accidentally upgrading or downgrading a package. Here is what Stefan Karpinski, co-creator of JuliaLang, explained to me in a recent exchange on Github:

There is no overriding of requirements and specific versions aren’t requested: there are compatibility bounds, often lower but sometimes upper; all requirements must be satisfied, including yours. When there’s some conflict, there tends to be more than one Pareto-optimal solution to the version selection problem. The resolver chooses one of these solutions, which uses a less recent version of some dependency, but it could have used a less recent version of some other dependency. If there was no conflict, it could just use the most recent version of all dependencies, but there is and it can’t. You can guide it to choose a different solution by requiring that the dependency that it chose the older version of be installed at a higher version (if that’s possible). The core issue here is that these version resolution problems are very complex (literally NP-hard) and it’s an open problem how to figure out and clearly present what the “core conflict” is. If we knew how to do that, we would. Even computing what all the possible optimal solutions are is hard and not something we can currently do.

1 Like
12

I have removed Mads now. I can’t quite remember why I ever added it.

Maybe it’s a fundamental package for the system, and I never added it myself…? If Mads is critical to other packages, and removing it will cause problems down the road, please let me know. Thanks!

13

Well, I’ve seemingly learned a lot more about package maintenance in the last day or two; it is more complicated than I had thought (hoped). My approach had been to load all the packages that I thought I might want to use, and have them available for all of my Julia programs & directories, everywhere. Apparently that is not a good idea.

I’m looking through the documentation, but one question I have now, is how to write permanent lines in the code (i.e., not at the command line or in the REPL), which tell the program to use the local directory environment, and not the global package environment.

I’m working on a Windows-based system, and I launch the program by clicking on the JuliaPro icon; I never launch it from the command line. Commands like activate ., which are run from the command line or in the REPL, are not commands that I want to have to interactively type every single time I open up JuliaPro, highlight a block of code, and run it. Are there non-interactive commands I can put at the top of the program (like, ex., using Plots), which I can permanently write into the program that tells it, every single time, “use the package environment in this local directory, not the global package environment”?

Also, when using Pkg.add("SomeGoodPackage"), I’m not sure what tells Julia to install it only to that local project directory, and not to the global package environment. Is typing "Pkg.activate ." beforehand enough to ensure that?

Thanks again…

P.S. ptoche, I’ve noticed that you’ve added a few lines of code in a previous comment; I vaguely see what you’re going for, but I’m not exactly sure what each of your commands does.

14

You just need something like

using Pkg; Pkg.activate(@__DIR__())

At the top of a file to make the file use an environment which will be created in the same folder as the file itself.

You are correct with regards to adding packages, they will get added to the Project.toml of whatever environment is currently active.

2 Likes
15 
    dir = expanduser("~/Julia/Experiments/")  # The directory of your package, for you maybe "C:\something"  
    push!(LOAD_PATH, dir)  # add the directory to the load path, so it can be found
    import Pkg  # or using Pkg
    Pkg.activate(dir)  # so now you activate the package
    Base.active_project()  # to make sure it's the package you meant to activate, print the path to console so you get a visual confirmation it's the package you meant to use
16

I was going to edit the typo in your thread title, but now I think “Sudden Falure”, has a nice ring to it, if you pronounce it right.

It’s also autological, I believe :wink:

1 Like