Optimization program which previously worked is no longer working

General Usage
, , ,
1

I am running julia version 1.10.5, about a month ago I had written and successfully run a piece of optimization code which uses LBFGS to optimize a system of 218 differential equations dependent on 12 parameters. Returning to this piece of code I am no longer able to run it (however it still works when run on smaller inputs). Since then my package environment has changed so perhaps this is the issue? I am unsure. IWhat are the possible causes and solutions to this issue? Now when I attempt to run the optimization code on the same input that worked a month ago I get the following error:

`[545371] signal (15): Terminated
in expression starting at /home/tighemcasey/StickySquaresProject/Optimize3x3/BestDeltasRun.jl:110
jl_gc_szclass at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/julia_internal.h:442 [inlined]
jl_gc_alloc_ at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/julia_internal.h:473 [inlined]
jl_gc_alloc at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gc.c:3583
ijl_new_bits at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/datatype.c:970
ijl_get_nth_field at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/datatype.c:1598
do_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/builtins.c:707

  • at ./operators.jl:587 [inlined]
    f1! at /home/tighemcasey/StickySquaresProject/Optimize3x3/Equations.jl:177
    Void at /home/tighemcasey/.julia/packages/SciMLBase/HkaWE/src/utils.jl:482 [inlined]
    CallWrapper at /home/tighemcasey/.julia/packages/FunctionWrappers/Q5cBx/src/FunctionWrappers.jl:65
    unknown function (ip: 0x7f1e5ed96404)
    macro expansion at /home/tighemcasey/.julia/packages/FunctionWrappers/Q5cBx/src/FunctionWrappers.jl:137 [inlined]
    do_ccall at /home/tighemcasey/.julia/packages/FunctionWrappers/Q5cBx/src/FunctionWrappers.jl:125 [inlined]
    FunctionWrapper at /home/tighemcasey/.julia/packages/FunctionWrappers/Q5cBx/src/FunctionWrappers.jl:144 [inlined]
    _call at /home/tighemcasey/.julia/packages/FunctionWrappersWrappers/9XR0m/src/FunctionWrappersWrappers.jl:12 [inlined]
    FunctionWrappersWrapper at /home/tighemcasey/.julia/packages/FunctionWrappersWrappers/9XR0m/src/FunctionWrappersWrappers.jl:10 [inlined]
    ODEFunction at /home/tighemcasey/.julia/packages/SciMLBase/HkaWE/src/scimlfunctions.jl:2335 [inlined]
    perform_step! at /home/tighemcasey/.julia/packages/OrdinaryDiffEqTsit5/DHYtz/src/tsit_perform_step.jl:204
    perform_step! at /home/tighemcasey/.julia/packages/OrdinaryDiffEqTsit5/DHYtz/src/tsit_perform_step.jl:181 [inlined]
    solve! at /home/tighemcasey/.julia/packages/OrdinaryDiffEqCore/4A2vD/src/solve.jl:551
    #__solve#61 at /home/tighemcasey/.julia/packages/OrdinaryDiffEqCore/4A2vD/src/solve.jl:7 [inlined]
    __solve at /home/tighemcasey/.julia/packages/OrdinaryDiffEqCore/4A2vD/src/solve.jl:1 [inlined]
    #solve_call#44 at /home/tighemcasey/.julia/packages/DiffEqBase/sCsah/src/solve.jl:612 [inlined]
    solve_call at /home/tighemcasey/.julia/packages/DiffEqBase/sCsah/src/solve.jl:569 [inlined]
    #solve_up#53 at /home/tighemcasey/.julia/packages/DiffEqBase/sCsah/src/solve.jl:1080 [inlined]
    solve_up at /home/tighemcasey/.julia/packages/DiffEqBase/sCsah/src/solve.jl:1066 [inlined]
    #solve#51 at /home/tighemcasey/.julia/packages/DiffEqBase/sCsah/src/solve.jl:1003 [inlined]
    solve at /home/tighemcasey/.julia/packages/DiffEqBase/sCsah/src/solve.jl:993
    unknown function (ip: 0x7f1e5edad93f)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    loss at /home/tighemcasey/StickySquaresProject/Optimize3x3/BestDeltasRun.jl:38
    #1 at /home/tighemcasey/StickySquaresProject/Optimize3x3/BestDeltasRun.jl:80 [inlined]
    #37 at /home/tighemcasey/.julia/packages/OptimizationBase/KIIy3/ext/OptimizationForwardDiffExt.jl:98 [inlined]
    #39 at /home/tighemcasey/.julia/packages/OptimizationBase/KIIy3/ext/OptimizationForwardDiffExt.jl:102 [inlined]
    vector_mode_dual_eval! at /home/tighemcasey/.julia/packages/ForwardDiff/PcZ48/src/apiutils.jl:24 [inlined]
    vector_mode_gradient! at /home/tighemcasey/.julia/packages/ForwardDiff/PcZ48/src/gradient.jl:96
    gradient! at /home/tighemcasey/.julia/packages/ForwardDiff/PcZ48/src/gradient.jl:37 [inlined]
    #38 at /home/tighemcasey/.julia/packages/OptimizationBase/KIIy3/ext/OptimizationForwardDiffExt.jl:102
    unknown function (ip: 0x7f1e5ef88369)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
    do_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/builtins.c:768
    #19 at /home/tighemcasey/.julia/packages/OptimizationOptimJL/hDX5k/src/OptimizationOptimJL.jl:297
    unknown function (ip: 0x7f1e5ef8436c)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    value_gradient!! at /home/tighemcasey/.julia/packages/NLSolversBase/kavn7/src/interface.jl:82
    value_gradient! at /home/tighemcasey/.julia/packages/NLSolversBase/kavn7/src/interface.jl:69
    value_gradient! at /home/tighemcasey/.julia/packages/Optim/ZhuZN/src/multivariate/solvers/constrained/fminbox.jl:94
    unknown function (ip: 0x7f1e5eef77f0)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    value_gradient! at /home/tighemcasey/.julia/packages/Optim/ZhuZN/src/Manifolds.jl:50
    ϕdϕ at /home/tighemcasey/.julia/packages/LineSearches/jgnxK/src/LineSearches.jl:83
    HagerZhang at /home/tighemcasey/.julia/packages/LineSearches/jgnxK/src/hagerzhang.jl:233
    HagerZhang at /home/tighemcasey/.julia/packages/LineSearches/jgnxK/src/hagerzhang.jl:102 [inlined]
    perform_linesearch! at /home/tighemcasey/.julia/packages/Optim/ZhuZN/src/utilities/perform_linesearch.jl:58
    update_state! at /home/tighemcasey/.julia/packages/Optim/ZhuZN/src/multivariate/solvers/first_order/l_bfgs.jl:204
    optimize at /home/tighemcasey/.julia/packages/Optim/ZhuZN/src/multivariate/optimize/optimize.jl:54
    optimize at /home/tighemcasey/.julia/packages/Optim/ZhuZN/src/multivariate/solvers/constrained/fminbox.jl:390
    unknown function (ip: 0x7f1e5ef83d6d)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    __solve at /home/tighemcasey/.julia/packages/OptimizationOptimJL/hDX5k/src/OptimizationOptimJL.jl:321
    solve! at /home/tighemcasey/.julia/packages/SciMLBase/HkaWE/src/solve.jl:188
    unknown function (ip: 0x7f1e5eed9a65)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    #solve#647 at /home/tighemcasey/.julia/packages/SciMLBase/HkaWE/src/solve.jl:96
    solve at /home/tighemcasey/.julia/packages/SciMLBase/HkaWE/src/solve.jl:93
    unknown function (ip: 0x7f1e5f06e5f0)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    yield_vs_Tmax at /home/tighemcasey/StickySquaresProject/Optimize3x3/BestDeltasRun.jl:83
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
    do_call at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/interpreter.c:126
    eval_value at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/interpreter.c:223
    eval_stmt_value at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
    eval_body at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/interpreter.c:617
    jl_interpret_toplevel_thunk at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/interpreter.c:775
    jl_toplevel_eval_flex at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/toplevel.c:934
    jl_toplevel_eval_flex at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/toplevel.c:877
    ijl_toplevel_eval_in at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/toplevel.c:985
    eval at ./boot.jl:385 [inlined]
    include_string at ./loading.jl:2076
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    _include at ./loading.jl:2136
    include at ./Base.jl:495
    jfptr_include_46447.1 at /snap/julia/104/lib/julia/sys.so (unknown line)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    exec_options at ./client.jl:318
    _start at ./client.jl:552
    jfptr__start_82798.1 at /snap/julia/104/lib/julia/sys.so (unknown line)
    _jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
    ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/gf.c:3077
    jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
    true_main at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/jlapi.c:582
    jl_repl_entrypoint at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/src/jlapi.c:731
    main at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
    unknown function (ip: 0x7f1f8e031d8f)
    __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
    unknown function (ip: 0x4010b8)
    unknown function (ip: (nil))
    Allocations: 46790740815 (Pool: 46438082700; Big: 352658115); GC: 59429`
2

Always make a backup copy of Manifest.toml when your code works to be able to reproduce the working environment if needed.

And it is hard to debug the exception you got without seeing your code and the output of using Pkg; Pkg.status()

1 Like
3

This would need reproducible code.

Are you using Julia v1.10.5? There’s GC fixes that are helpful. If you are on v1.10.5, are you using @inbounds anywhere in your code? What’s at that line of code?

4

Here is the output of Pkg.installed():

"CSV"                   => v"0.10.14"
  "DifferentialEquations" => v"7.13.0"
  "DataFrames"            => v"1.6.1"
  "Optimization"          => v"3.27.0"
  "SciMLSensitivity"      => v"7.65.0"
  "ChangePrecision"       => v"1.1.0"
  "OptimizationOptimJL"   => v"0.3.2"

Here is the optimization code, the file Constants.txt simply contains the numerical constants, f1! assigns the value of dC as a non-linear function of deltas and C.


#=
THIS FILE PRODUCES OPTIMAL BOND ENERGIES AND YIELDS AT THOSE BOND ENERGIES AT A GIVEN SET OF TIMESTEPS,
OUTPUT IS TO FILES 'YieldvsTime.txt' AND 'BindingEnergyvsTime.txt', 
NOTE THAT THE OUTPUT OF BINDING ENERGIES IS IN LEXICOGRAPHICAL ORDER
=#

# ------------------------------------ I/O and Inclusions ------------------------------------ #

# Setup Files
include("Equations.jl")
constantspath = joinpath(@__DIR__,"Constants.txt")

# Output Files
Yieldpath = joinpath(@__DIR__,"YieldvsTime.txt")
Bindingpath = joinpath(@__DIR__,"BindingEnergyvsTime.txt")

#Read Necessary Parameters from 'Constants.txt'
params = readdlm(constantspath,skipstart=2)
concentration = params[4,1]::Float64
numblocks = params[7,1]::Int64
numEdges = params[2,1]::Int64
numVertices = params[3,1]::Int64
C = Vector{Float64}(params[5,1:numblocks])
dC = Vector{Float64}(params[6,1:numblocks])

# Write output to file
function writeoutput(Yieldpath, Bindingpath, concvec, deltaarray)
    #Write Header
    maxtime = tvec[end]::Float64
    deltastring = ""
    for k in 1:numEdges
        deltastring = deltastring*"-delta$k:  "
    end
    yieldheader = "density = $concentration ; Initial deltas = $deltas ; \n Initial concentrations = $C ;\
     max time = $maxtime \n final C(Target Structure):  maxtime: \n"
    bindingheader = "density = $concentration ; Initial deltas = $deltas ; \n Initial concentrations = $C ;\
     max time = $maxtime \n"*deltastring*" maxtime: \n"
    write(Yieldpath,yieldheader)
    write(Bindingpath, bindingheader)

    #Write Data
    CSV.write(Yieldpath,Tables.table(concvec),append=true)
    CSV.write(Bindingpath,Tables.table(deltaarray),append=true)
end


# ------------------------------------ Input Parameters ------------------------------------ #

tvec = [1000.0,10000.0,100000.0] # Time Steps Optimized At
deltas = [-0.3546379129258437, -1.0306384019336112, -2.765023007323766, -2.5384636391235658, -1.816160723724639, -6.0154802607957905, -4.734737685358027, -5.7959495615945364, -0.5149777996841498, -3.536189921414051, -9.460076287814163, -2.9729010182065396]
tspan = (0.0,100.0) # Time span for initial solve
maxenergy = -15.0 # Boundary condition on bonding energies

#Parameters for ODE solver and optimization
adtype = Optimization.AutoForwardDiff()
maxoptiterations = 200

#define ode problem
problem = ODEProblem(f1!,C,tspan,deltas)
sol = solve(problem,save_on = false)


# ------------------------------------ Optimization ------------------------------------ #

#Loss function
function loss(newp,t)
    newproblem = remake(problem,p = newp, tspan = (0.0,t))
    sol = solve(newproblem, Tsit5(), save_on = false)
    if Sys.free_memory() / 2^30 < 15 #Possibly helps with memory issues
        GC.gc()
    end
    loss = -sol[end,end] #Final concentration of target structure
    return loss, sol
end

function yield_vs_Tmax(tvec,problem,adtype)
    #Initialize storage for data
    deltaarray = Array{Float64}(undef,size(tvec)[1],numEdges+1)
    concvec = Array{Float64}(undef,size(tvec)[1],2)
    i = 1 #Used for indexing
    #Optimization code for each Tmax:
    for t ∈ tvec
        optf = Optimization.OptimizationFunction((x, p) -> loss(x,t), adtype) #Define optimization loss function, in this case maximizes concentration of final structure
        optprob = Optimization.OptimizationProblem(optf, deltas, lb = ones(numEdges).*maxenergy, ub = zeros(numEdges)) #define problem and boundary conditions
        result_ode = Optimization.solve(optprob, LBFGS(), maxiters = maxoptiterations)

        for j ∈ 1:numEdges+1
            if j ≤ numEdges
                deltaarray[i,j] = -result_ode.u[j]
            else
                #deltaarray[i,:] = sort(deltaarray[i,:],rev=true) #In the case of symmetry a sorted output might make more sense
                deltaarray[i,j] = t
            end
        end
        concvec[i,1] = -result_ode.objective*numVertices/concentration
        concvec[i,2] = t
        i += 1
        println("Done optimization for t = $t")
    end
    #Write results to file
    writeoutput(Yieldpath,Bindingpath, concvec, deltaarray)
end

println("Started Parameter Optimization")
yield_vs_Tmax(tvec,problem,adtype)
println("Completed Parameter Optimization")```
5

I am not using @inline anywhere.

6

And Julia v1.10.5?

7

Yes

8

and?

9

@inline and @inbounds do two very different things, but I’m assuming you do neither. You can start Julia with --check-bounds=yes to confirm that it isn’t some other package using @inbounds causing the problem.

10

Here is the equations function; it is over the character limit substantially

Equations.jl (145.9 KB)

12

yes, I use neither. I will try running the program with this startup option.

13

The program has the same error when run with these options.

14

What’s the reproducer and the error?