Simple SciML case - core dumped after switching from julia 1.7 to 1.8

General Usage
1

Hi there, I am starting with SciML and getting a core dumped when switching from julia 1.7.2 to julia 1.8.1

It is a simple combination of ODEProblem and Optimization, worked before, now fails dramatically.

import DifferentialEquations, DiffEqSensitivity, DiffEqFlux
import Symbolics, ModelingToolkit, DataDrivenDiffEq
import Optimization, OptimizationOptimisers, OptimizationOptimJL
import Lux, ComponentArrays
import Plots, Random, Statistics, StatsBase, DelimitedFiles

using CommonSolve: solve

rng = Random.default_rng()
Random.seed!(rng, 12345)

function lotka_volterra(du, u, p, t)
    x, y = u
    α, β, δ, γ = p
    du[1] = dx = α*x - β*x*y
    du[2] = dy = -δ*y + γ*x*y
end
u0 = [1.0, 1.0]
tspan = (0.0, 10.0)
p = [1.5, 1.0, 3.0, 1.0]
ode_prob = DifferentialEquations.ODEProblem(lotka_volterra, u0, tspan, p)
ode_sol = solve(ode_prob, saveat=0.1)

function predict(parameters, ode_prob=ode_prob, t=ode_sol.t)
    solve(ode_prob, saveat = t, p = parameters)
end
function loss_function(parameters, data)
    pred = Array(predict(parameters))[1,:]
    return sum(abs2, pred .- data)
end

ps_initial = ode_prob.p
data = 1.0
loss_function(ps_initial, data)

losses = Float64[]
function callback(p, l)
    push!(losses, l)
    if length(losses) % 50 == 0
        Plots.plot(losses, show = :inline, yscale = :log10,
            label = "loss", xlabel = "#epochs", ylabel="loss (log10 scale)")
    end
    return false  # return bool `halt`
end

ps_trained = let data = data
    minimizer = ps_initial
    opt_function = Optimization.OptimizationFunction(
        (ps, data) -> loss_function(ps, data),
        Optimization.AutoZygote(),
    )
    for (optimizer, maxiters) = [
            (OptimizationOptimisers.Adam(0.1), 300),
            (OptimizationOptimisers.Adam(0.01), 500),
        ]
        opt_prob = Optimization.OptimizationProblem(opt_function, minimizer, data)
        opt_sol = solve(opt_prob, optimizer,
            callback = callback, maxiters = maxiters)
        minimizer = opt_sol.minimizer
    end
    minimizer
end

This will throw julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.h:2093: llvm::SmallVector<llvm::SelectInst*, 4> DiffeGradientUtils::addToDiffe(llvm::Value*, llvm::Value*, llvm::IRBuilder<>&, llvm::Type*, llvm::ArrayRef<llvm::Value*>, llvm::Value*): Assertion !isConstantValue(val)’ failed.`

here all the details:

; Function Attrs: mustprogress willreturn
define internal void @diffejulia__220_12422_inner.1([1 x { { [1 x [4 x {} addrspace(10)*]], [1 x i8] }, {} addrspace(10)*, [2 x double], {} addrspace(10)* }] %0, {} addrspace(10)* nonnull align 16 dereferenceable(40) %1, {} addrspace(10)* %"'", {} addrspace(10)* nonnull align 16 dereferenceable(40) %2, {} addrspace(10)* nonnull align 16 dereferenceable(40) %3, double %4) local_unnamed_addr #9 !dbg !109 {
entry:
%.fca.0.0.0.0.0.extract = extractvalue [1 x { { [1 x [4 x {} addrspace(10)*]], [1 x i8] }, {} addrspace(10)*, [2 x double], {} addrspace(10)* }] %0, 0, 0, 0, 0, 0, !dbg !110
%_replacementA = phi {}*** 
call void @llvm.assume(i1 noundef true) #14
%5 = bitcast {} addrspace(10)* %.fca.0.0.0.0.0.extract to i64 addrspace(10)*, !dbg !111
%6 = addrspacecast i64 addrspace(10)* %5 to i64 addrspace(11)*, !dbg !111
%7 = load i64, i64 addrspace(11)* %6, align 8, !dbg !111, !tbaa !38
%.not = icmp eq i64 %7, 0, !dbg !120
br i1 %.not, label %L12.i, label %L13.i, !dbg !123

L12.i:                                            ; preds = %entry
%_augmented = call fastcc { { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }, i64, i64 } @augmented_julia_reinit_wrapper_12426({} addrspace(10)* %.fca.0.0.0.0.0.extract), !dbg !124
%subcache = extractvalue { { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }, i64, i64 } %_augmented, 0, !dbg !124
store { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* } %subcache, { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }* %subcache_cache, align 1, !dbg !124, !invariant.group !125
%8 = extractvalue { { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }, i64, i64 } %_augmented, 1, !dbg !124
%"'ac" = extractvalue { { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }, i64, i64 } %_augmented, 2, !dbg !124
br label %L13.i, !dbg !124

L13.i:                                            ; preds = %L12.i, %entry
%9 = phi i64 [ %"'ac", %L12.i ], [ %7, %entry ]
%value_phi.i = phi i64 [ %8, %L12.i ], [ %7, %entry ]
call fastcc void @julia__220_12422u12425() #15, !dbg !126
%.not7 = icmp eq i64 %value_phi.i, 0, !dbg !128
br i1 %.not7, label %fail.i, label %julia__220_12422_inner.exit, !dbg !128

fail.i:                                           ; preds = %L13.i
call void @ijl_throw({} addrspace(12)* noundef addrspacecast ({}* inttoptr (i64 140567845548720 to {}*) to {} addrspace(12)*)) #16, !dbg !128
unreachable

julia__220_12422_inner.exit:                      ; preds = %L13.i
%10 = bitcast {} addrspace(10)* %.fca.0.0.0.0.0.extract to i8 addrspace(10)*, !dbg !129
%11 = addrspacecast i8 addrspace(10)* %10 to i8 addrspace(11)*, !dbg !129
%12 = getelementptr inbounds i8, i8 addrspace(11)* %11, i64 8, !dbg !129
%13 = bitcast i8 addrspace(11)* %12 to i64 addrspace(11)*, !dbg !129
%14 = load i64, i64 addrspace(11)* %13, align 8, !dbg !129, !tbaa !38
%"'il_phi1" = phi i64 , !dbg !128
%"'ipc" = inttoptr i64 %9 to void (i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)*, !dbg !128
%15 = inttoptr i64 %value_phi.i to void (i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)*, !dbg !128
%16 = bitcast void (i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)* %"'ipc" to { i8* } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)**, !dbg!128
%17 = load { i8* } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)*, { i8* } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)** %16, align 8, !dbg !128
%_augmented2 = call { i8* } %17(i64 %14, i64 %14, {} addrspace(10)* %1, {} addrspace(10)* %"'", {} addrspace(10)* %2, {} addrspace(10)* %2, {} addrspace(10)* %3, {} addrspace(10)* %3, double %4) [ "jl_roots"({} addrspace(10)* %3, {} addrspace(10)* %2, {} addrspace(10)* %1, {} addrspace(10)* %"'") ], !dbg !128
%subcache3 = extractvalue { i8* } %_augmented2, 0, !dbg !128
br label %invertjulia__220_12422_inner.exit, !dbg !110

allocsForInversion:                               ; No predecessors!
%subcache_cache = alloca { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }, align 1
store { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* } zeroinitializer, { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }* %subcache_cache, align 8

invertentry:                                      ; preds = %invertL13.i, %invertL12.i
ret void

invertL12.i:                                      ; preds = %invertL13.i
%18 = load { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }, { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* }* %subcache_cache, align 1, !invariant.group !125
call fastcc void @diffejulia_reinit_wrapper_12426({} addrspace(10)* %.fca.0.0.0.0.0.extract, { {} addrspace(10)*, {} addrspace(10)*, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)* } %18), !dbg !124
br label %invertentry

invertL13.i:                                      ; No predecessors!
br i1 %.not, label %invertL12.i, label %invertentry

invertfail.i:                                     ; No predecessors!

invertjulia__220_12422_inner.exit:                ; preds = %julia__220_12422_inner.exit
%19 = bitcast void (i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double)* %"'ipc" to { double } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double, i8*)**
%20 = getelementptr { double } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double, i8*)*, { double } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double, i8*)** %19, i64 1
%21 = load { double } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double, i8*)*, { double } (i64, i64, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double, i8*)** %20, align 8
%22 = call { double } %21(i64 %14, i64 %14, {} addrspace(10)* %1, {} addrspace(10)* %"'", {} addrspace(10)* %2, {} addrspace(10)* %2, {} addrspace(10)* %3, {} addrspace(10)* %3, double %4, i8* %subcache3) [ "jl_roots"({} addrspace(10)* %3, {} addrspace(10)* %2, {} addrspace(10)* %1, {} addrspace(10)* %"'") ], !dbg !128
%23 = extractvalue { double } %22, 0
}

double %4
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.h:2093: llvm::SmallVector<llvm::SelectInst*, 4> DiffeGradientUtils::addToDiffe(llvm::Value*, llvm::Value*, llvm::IRBuilder<>&, llvm::Type*, llvm::ArrayRef<llvm::Value*>, llvm::Value*): Assertion `!isConstantValue(val)' failed.

signal (6): Aborted
in expression starting at REPL[30]:1
__pthread_kill_implementation at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
raise at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
abort at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
__assert_fail_base.cold.0 at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
__assert_fail at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
addToDiffe at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.h:2093
addToDiffe at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:1754 [inlined]
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:12285
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:302 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:112 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:3646
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:439
EnzymeCreatePrimalAndGradient at /home/ssahm/.julia/packages/Enzyme/di3zM/src/api.jl:111
enzyme! at /home/ssahm/.julia/packages/Enzyme/di3zM/src/compiler.jl:3271
unknown function (ip: 0x7fd83e51f92d)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#codegen#80 at /home/ssahm/.julia/packages/Enzyme/di3zM/src/compiler.jl:4158
codegen##kw at /home/ssahm/.julia/packages/Enzyme/di3zM/src/compiler.jl:3878 [inlined]
_thunk at /home/ssahm/.julia/packages/Enzyme/di3zM/src/compiler.jl:4599
unknown function (ip: 0x7fd83e8d422d)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
cached_compilation at /home/ssahm/.julia/packages/Enzyme/di3zM/src/compiler.jl:4637
unknown function (ip: 0x7fd83e8949d5)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#s565#115 at /home/ssahm/.julia/packages/Enzyme/di3zM/src/compiler.jl:4697 [inlined]
#s565#115 at ./none:0
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
GeneratedFunctionStub at ./boot.jl:582
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_call_staged at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/method.c:520
ijl_code_for_staged at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/method.c:571
get_staged at ./compiler/utilities.jl:114
retrieve_code_info at ./compiler/utilities.jl:126 [inlined]
InferenceState at ./compiler/inferencestate.jl:284
typeinf_edge at ./compiler/typeinfer.jl:868
abstract_call_method at ./compiler/abstractinterpretation.jl:641
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1696
abstract_call at ./compiler/abstractinterpretation.jl:1766
abstract_call at ./compiler/abstractinterpretation.jl:1733
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1890
typeinf_local at ./compiler/abstractinterpretation.jl:2366
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2462
_typeinf at ./compiler/typeinfer.jl:230
typeinf at ./compiler/typeinfer.jl:213
[... for the full log see https://github.com/SciML/Optimization.jl/issues/372 ...] 
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#967 at ./client.jl:419
jfptr_YY.967_49700.clone_1 at /nix/store/ca4hhym3f57vpmhgvylvqp86cmz9gbis-julia-bin-1.8.1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:729 [inlined]
invokelatest at ./essentials.jl:726 [inlined]
run_main_repl at ./client.jl:404
exec_options at ./client.jl:318
_start at ./client.jl:522
jfptr__start_61720.clone_1 at /nix/store/ca4hhym3f57vpmhgvylvqp86cmz9gbis-julia-bin-1.8.1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
true_main at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci5-0/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at julia (unknown line)
__libc_start_call_main at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
__libc_start_main at /nix/store/bzd91shky9j9d43girrrj6vmqlw7x9m8-glibc-2.35-163/lib/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 401837661 (Pool: 401721260; Big: 116401); GC: 101
[1]    441621 abort (core dumped)  julia --project

Here my manifest.toml (as txt because github won’t allow .toml files):
Manifest.toml.txt

I already raised this as an issue on Optimization.jl Simple SciML case - core dumped after switching from julia 1.7 to 1.8 · Issue #372 · SciML/Optimization.jl · GitHub but it occured to me that the problem might be somewhere completely different, so probably it is more apt for a discourse issue.

Any help is highly appreciated

2

There’s no need to repost the same issue multiple times in a day. I’ll get to it, I was just writing a blog post yesterday. Blog post on compile time improvements by ChrisRackauckas · Pull Request #90 · SciML/sciml.ai · GitHub . More emails of the same thing won’t change time.

Let’s keep this discussion to the equivalent issue.

3

Thank you for the clarification and sorry for the noise.

4

It looks like the issue was using the older DiffEqSensitivity v6 instead of the updated SciMLSensitivity v7. All sources should be updated to SciMLSensitivity.jl when possible, as DiffEqSensitivity was deprecated earlier this year. If you find any tutorials with the old library, please let me know and I’ll update it.

1 Like
5

I think I found the old tutorial. The Optimization.jl documentation was still showing an older v3.8.2 version of the documentation from June. I just setup a tag to release v3.9 which should update that tutorial.

6

Data Iterators and Minibatching · Optimization.jl is updated on stable now. Sorry about that.

1 Like
7

Thank you so much about your understanding and help. It works now with SciMLSensitivity :slight_smile:

Your help and commitment to SciML is impressive. You have my admiration.

Because you asked, I also used the code for the original UDE paper as a start: universal_differential_equations/LotkaVolterra at master · ChrisRackauckas/universal_differential_equations · GitHub, especially universal_differential_equations/scenario_2.jl at master · ChrisRackauckas/universal_differential_equations · GitHub which was updated 2 months ago. Hence I thought they are trustworthy and (at least in parts) kept up-to-date.

Would be really awesome if those example could be kept up-to-date such that they can be taken as an alternative start to get into SciML UDEs. (in case that matters, you can point to the original code in the front readme by linking the github-repo-fixed-at-a-specific-commit).

8

Those are not up to date. My plan is to get versions of those into the SciMLSensitivity.jl documentation as tutorials, so that way they get tested along with the release process. If they are in a separate untested repo, they will continually go out of date, as they do now, which is a sad problem. I planned to do that this month but got a bit behind on it, so there was some version bump but not the complete one IIRC (there might still be an open PR with a better version?)

1 Like