Debugging an anonymous function passed to DynamicalSystems

HI. This question is a continuation of a previous topic here, on constructing anonymous functions dynamically. In my case, these functions are intended to form the dynamical rule used for initialising CoupledODEs from a set of stock, flow and variable specifications entered by the user.

Here is the sample code, as far as I have been able to strip it down for this question. Please note that the SD (system dynamics) model rabbits (created in the method demo() at the bottom) could contain arbitrarily many stocks, variables and flows. Also variables could contain arithmetic expressions, as well as (in this case) the numerical value 0.3:

module SD

using DynamicalSystems

#-----------------------------------------------------------------------------------------
# Module types:
#-----------------------------------------------------------------------------------------
# VExpr represents anything that can be eval()ed to a Number: Exprs, identifiers and Numbers.
VExpr = Union{Expr,Symbol,Number}

mutable struct SDModel
	stocks::Dict{Symbol,Float64}
	flows::Dict{Symbol,Vector{VExpr}}
	variables::Dict{Symbol,VExpr}
	ode_model::Union{Nothing,CoupledODEs}

	function SDModel( title="An Exponential SD model")
		new( Dict{Symbol,Float64}(),
			Dict{Symbol,Vector{VExpr}}(),
			Dict{Symbol,VExpr}(),
			nothing
		)
	end
end

#-----------------------------------------------------------------------------------------
# Module methods:
#-----------------------------------------------------------------------------------------
"""
Build the CoupledODEs required for the SDModel to run.
"""
function build!( sdm)
	p_keys = collect(keys(sdm.variables))
	p_vals = deepcopy(collect(values(sdm.variables)))
	u_keys = collect(keys(sdm.stocks))
	u_vals = collect(values(sdm.stocks))
	du_vals = deepcopy(collect(values(sdm.flows)))
	p_builder = Vector{Function}(undef,length(p_keys))
	du_builder = Vector{Function}(undef,length(u_keys))

	# Build dynamical rules for parameters:
	for (j,expr) in enumerate(p_vals)
		for (i,k) in enumerate(p_keys)
			substitute!(expr, k, :(p[$i]))
		end
		for (i,k) in enumerate(u_keys)
			substitute!(expr, k, :(u[$i]))
		end
		println("!!! Dump of function to update p:")
		dump(Expr(:(->),:(p,u),expr))
		p_builder[j] = eval(Expr(:(->),:(p,u),expr))
	end

	# Build dynamical rules for derivatives:
	for (j,flow_set) in enumerate(du_vals)
		for (i,k) in enumerate(p_keys)
			for flow_term in flow_set
				substitute!(flow_term,k,:(p[$i]))
			end
		end
		for (i,k) in enumerate(u_keys)
			for flow_term in flow_set
				substitute!(flow_term,k,:(u[$i]))
			end
		end
		println("\n!!! Dump of function to update du:")
		dump(Expr(:(->),:(p,u),Expr(:call,:(+),flow_set...)))
		du_builder[j] = eval(Expr(:(->),:(p,u),Expr(:call,:(+),flow_set...)))
	end

	# Build general dynamical rule for CoupledODEs :
	dynamical_rule! = function (du,u,p,t)
		for i in eachindex(p)
			p[i] = Float64(p_builder[i](p,u))
		end
		for i in 1:length(u_keys)
			du[i] = Float64(eval(du_builder[i](p,u)))
		end
		nothing
	end

	# Alternative dynamical_rule! that is capable of running:
#=	dynamical_rule! = function (du,u,p,t)
		p[1] = 0.3
		du[1] = p[1] * u[1]
		nothing
	end=#

	sdm.ode_model = CoupledODEs(dynamical_rule!, u_vals, p_vals)
end

#-----------------------------------------------------------------------------------------
"""
Substitute IN PLACE each occurrence of old_argument by the new_arg in the given expression.
"""
function substitute!( expr::Expr, old_arg, new_arg)
	for (i,arg) in enumerate(expr.args)
		if arg==old_arg
			expr.args[i] = new_arg
		elseif arg isa Expr
			substitute!(arg, old_arg, new_arg)
		end
	end

	nothing
end

function substitute!( sym::Symbol, old_arg, new_arg)
	if old_arg == sym
		sym = new_arg
	end

	nothing
end

substitute!( expr::Number, old_arg, new_arg) = nothing

#-----------------------------------------------------------------------------------------
"""
Use-case for building a simple exponential model.
"""
function demo()
	rabbits = SDModel()
	rabbits.stocks[:Rabbits] = 2.0
	rabbits.variables[:mu] = Meta.quot(0.3)
	rabbits.flows[:Rabbits] = [Meta.parse("mu * Rabbits")]
	println()
	println( "!!! Dump of pre-build rabbits model:")
	display(rabbits)
	println()
	build!( rabbits)
	println()
	trajectory(rabbits.ode_model, 3; Δt=0.25)
end

end

If this code is run with the alternative dynamical_rule! commented out in build!(), it runs correctly. If, however, I run it with the currently used dynamical_rule! constructed within the build!(), this rule method appears to do its job correctly in setting the values of p and du, yet when I pass it to the CoupledODEs constructor, this throws errors connected to the fact that I use the union type VExpr = Union{Expr, Number, Symbol} to manage expression syntax.

My question is this: I feel I could solve these issues if only I could get a handle on what precisely is the nature of the internal problem for CoupledODEs(). However, I’m finding it very difficult to obtain that information. Any ideas?

When you say “problem” do you refer to the DifferentialEquations.jl DEProblem type? That is just ODEProblem.

If the information you need is the one above, then this shouldn’t be hard to find. Just typing @edit CoupledODEs(...) would bring you to https://github.com/JuliaDynamics/DynamicalSystemsBase.jl/blob/210cd2d3311c2f2a0b4b682dae9d53739f1b8b54/src/core_systems/continuous_time_ode.jl#L79-L85 which shows you the ODEProblem.

(I haven’t read the full message yet, only the last paragraph, sorry short on time!)

Sorry, no - I didn’t express myself clearly. By ‘problem’, I mean: What is it about my constructed anonymous function dynamical_rule! that CoupledODEs finds so troublesome? As far as I can make out, dynamical_rule! doesn’t make any bothersome type demands on its arguments, and it does its job of setting du correctly, so why the execution error?

Of, course I am aware that there is something rather suspect about the fact that I set the values of p within this dynamical rule, and I’ll need to find out how I should be doing that properly, but for the moment, the fact that execution runs ok for the simplified (commented out) dynamical rule at the end of build!() means that this is not the issue.

I’m afraid I’m totally flummoxed. :confused:

You cannot be changing the parameters of the system within the dynamical rule. The very definition of a parameter is that it does not change in time. Everything that is changing inside the dynamical rule must be a state variable, i.e., a component of u. You need to adjust your logical approach to the problem accordingly.

Alternatively, you can change system parameters in-between steps of the model. This can be done either via the callback system of DifferentialEquations.jl (Event Handling and Callback Functions · DifferentialEquations.jl ) or by simply calling the function set_parameter! inbetween calls to step!(ds::DynamicalSystem, t::Real).

Great - thanks. That clears up the second question I had regarding how to update function values. So what would be your recommendation for updating a function value something like v = sin(u[1]): should I implement it as a parameter p[j] that is updated between steps, or as an additional state variable u[j] = sin(u[1])?

However, to return to my original question, this cannot be the cause of the runtime error, can it? Because if I replace the current dynamical_rule! by the one that is currently commented out (which also sets the value of p[1] to 0.3), CoupledODEs seems to be quite happy with it.

Yes, for that I would advise you to make a Minimal Working Example, that also includes explicitly the error message and the stacktrace, as the code you have pasted is unecessarily extensive and I can’t go through it at the moment.

:grinning: Yup, fair enough. I’d already scrapped a ton of code, but you’re right - what’s left still isn’t minimal. Just note for the moment that dynamical_rule! is the assembled result of a load of substitution and manipulation of Exprs, and I’ll make sure I get out a minimal example by the end of the day. Thanks very much! :grin:

OK, here it is. And boiled down like this, it seems that the problem lies neither in DynamicalSystems nor in the automatically assembled dynamical rule. The following code runs perfectly:

using DynamicalSystems

#function demo()
	dub = :(+(p[1] * u[1]))
	du_builder = eval(Expr(:(->),:(p,u),dub))

	# Build general dynamical rule for CoupledODEs :
	dynamical_rule! = function (du,u,p,t)
		du[1] = Float64(eval(du_builder(p,u)))
		nothing
	end

	du,u,p,t = ([1.0],[2.0],[0.3],4.0)
	println("du_builder does the right job - stores the value p[1]*u[1]=0.6 in du[1]:")
	println("Values before running dynamical_rule!: (du,u,p,t) = $((du,u,p,t))")
	dynamical_rule!(du,u,p,t)
	println("Values after  running dynamical_rule!: (du,u,p,t) = $((du,u,p,t))")

	ode_model = CoupledODEs(dynamical_rule!, [2.0], [0.3])
	trajectory(ode_model, 3; Δt=0.25)[1][:]
#end

#demo()

Yet if I remove the comment markers, the code no longer runs. I should add that the code then contains two errors - one when dynamical_rule! is explicitly called, but if we comment out that line, CoupledODEs is also unhappy.

Once again, I appear to be misunderstanding something fundamental here.

I’m sorry - I’ve only just realised that you explicitly requested the error message and stack trace. I have reduced the sample code to the following:

module SD

using DynamicalSystems

function demo()
	builder = eval(Expr(:(->),:(p,u),:(+(p[1] * u[1]))))
	
	dynamical_rule! = function (du,u,p,t)
		du[1] = Float64(eval(builder(p,u)))
		nothing
	end

	CoupledODEs(dynamical_rule!, [2.0], [0.3])
end
end

Calling SD.demo() leads to an error message whose essence is: The applicable method may be too new: running in world age 33731, while current world is 33732. (I list the entire stack trace below)

I now understand the reasons for this error in terms of julia ensuring type stability of dynamical_rule!, however I’m starting to suspect that solving this problem is impossible, since it would presumably involve DynamicalSystems internally calling dynamical_rule! using invokelatest(), which would then hugely impact performance.

So my final question is: Is there any way at all of constructing an anonymous function at runtime which can then be passed as a dynamical rule to DynamicalSystems? I realise that one fix is to pull the first line of demo() (the definition of builder) out into the module’s global scope, but I don’t see that as viable in the context of dynamically parsing users’ function specifications.

The complete trace from the call to SD.demo() is:

ERROR: MethodError: no method matching (::Main.SD.var"#3#4")(::Vector{Float64}, ::Vector{Float64})
The applicable method may be too new: running in world age 33731, while current world is 33732.   

Closest candidates are:
  (::Main.SD.var"#3#4")(::Any, ::Any) (method too new to be called from this world context.)
   @ Main.SD none:0

Stacktrace:
  [1] (::Main.SD.var"#1#2"{Main.SD.var"#3#4"})(du::Vector{Float64}, u::Vector{Float64}, p::Vector{Float64}, t::Float64)
    @ Main.SD c:\Users\hswt136nia\OneDrive\Documents\Rock Dene Cottage\Courses\LEAP\Projects\Anatta\src\Development\SD\SD.jl:9
  [2] (::SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing})(::Vector{Float64}, ::Vararg{Any})
    @ SciMLBase C:\Users\hswt136nia\.julia\packages\SciMLBase\eK30d\src\scimlfunctions.jl:2407
  [3] initialize!(integrator::OrdinaryDiffEq.ODEIntegrator{OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, true, Vector{Float64}, Nothing, Float64, Vector{Float64}, Float64, Float64, Float64, Float64, Vector{Vector{Float64}}, SciMLBase.ODESolution{Float64, 2, Vector{Vector{Float64}}, Nothing, Nothing, Vector{Float64}, Vector{Vector{Vector{Float64}}}, 
SciMLBase.ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, 
Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.InterpolationData{SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Vector{Vector{Float64}}, Vector{Float64}, Vector{Vector{Vector{Float64}}}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}}, SciMLBase.DEStats, Nothing}, SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, 
Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, OrdinaryDiffEq.DEOptions{Float64, Float64, Float64, Float64, OrdinaryDiffEq.PIController{Rational{Int64}}, typeof(DiffEqBase.ODE_DEFAULT_NORM), typeof(LinearAlgebra.opnorm), Bool, SciMLBase.CallbackSet{Tuple{}, Tuple{}}, typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, DataStructures.BinaryHeap{Float64, DataStructures.FasterForward}, Nothing, Nothing, Float64, Tuple{}, Tuple{}, Tuple{}}, Vector{Float64}, Float64, Nothing, OrdinaryDiffEq.DefaultInit}, cache::OrdinaryDiffEq.Tsit5Cache{Vector{Float64}, Vector{Float64}, Vector{Float64}, typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False})
    @ OrdinaryDiffEq C:\Users\hswt136nia\.julia\packages\OrdinaryDiffEq\JJd6g\src\perform_step\low_order_rk_perform_step.jl:792
  [4] __init(prob::SciMLBase.ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, alg::OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, timeseries_init::Tuple{}, ts_init::Tuple{}, ks_init::Tuple{}, recompile::Type{Val{true}}; saveat::Tuple{}, tstops::Tuple{}, d_discontinuities::Tuple{}, save_idxs::Nothing, save_everystep::Bool, save_on::Bool, save_start::Bool, save_end::Bool, callback::Nothing, dense::Bool, calck::Bool, dt::Float64, dtmin::Nothing, dtmax::Float64, force_dtmin::Bool, adaptive::Bool, gamma::Rational{Int64}, abstol::Float64, reltol::Float64, qmin::Rational{Int64}, qmax::Int64, qsteady_min::Int64, qsteady_max::Int64, beta1::Nothing, beta2::Nothing, qoldinit::Rational{Int64}, controller::Nothing, fullnormalize::Bool, failfactor::Int64, maxiters::Float64, internalnorm::typeof(DiffEqBase.ODE_DEFAULT_NORM), internalopnorm::typeof(LinearAlgebra.opnorm), isoutofdomain::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), unstable_check::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), verbose::Bool, timeseries_errors::Bool, dense_errors::Bool, advance_to_tstop::Bool, stop_at_next_tstop::Bool, initialize_save::Bool, progress::Bool, progress_steps::Int64, progress_name::String, progress_message::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), progress_id::Symbol, userdata::Nothing, allow_extrapolation::Bool, initialize_integrator::Bool, alias_u0::Bool, alias_du0::Bool, initializealg::OrdinaryDiffEq.DefaultInit, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ OrdinaryDiffEq C:\Users\hswt136nia\.julia\packages\OrdinaryDiffEq\JJd6g\src\solve.jl:502
  [5] __init (repeats 4 times)
    @ C:\Users\hswt136nia\.julia\packages\OrdinaryDiffEq\JJd6g\src\solve.jl:10 [inlined]
  [6] DynamicalSystemsBase.CoupledODEs(prob::SciMLBase.ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, diffeq::NamedTuple{(:alg, :abstol, :reltol), Tuple{OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, Float64, Float64}}; special_kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ DynamicalSystemsBase C:\Users\hswt136nia\.julia\packages\DynamicalSystemsBase\Dx8Dp\src\core_systems\continuous_time_ode.jl:93
  [7] DynamicalSystemsBase.CoupledODEs(prob::SciMLBase.ODEProblem{Vector{Float64}, Tuple{Float64, Float64}, true, Vector{Float64}, SciMLBase.ODEFunction{true, SciMLBase.AutoSpecialize, Main.SD.var"#1#2"{Main.SD.var"#3#4"}, LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, diffeq::NamedTuple{(:alg, :abstol, :reltol), Tuple{OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, Float64, Float64}})
    @ DynamicalSystemsBase C:\Users\hswt136nia\.julia\packages\DynamicalSystemsBase\Dx8Dp\src\core_systems\continuous_time_ode.jl:88
  [8] DynamicalSystemsBase.CoupledODEs(f::Function, u0::Vector{Float64}, p::Vector{Float64}; t0::Int64, diffeq::NamedTuple{(:alg, :abstol, :reltol), Tuple{OrdinaryDiffEq.Tsit5{typeof(OrdinaryDiffEq.trivial_limiter!), typeof(OrdinaryDiffEq.trivial_limiter!), Static.False}, Float64, 
Float64}})
    @ DynamicalSystemsBase C:\Users\hswt136nia\.julia\packages\DynamicalSystemsBase\Dx8Dp\src\core_systems\continuous_time_ode.jl:84
  [9] CoupledODEs
    @ C:\Users\hswt136nia\.julia\packages\DynamicalSystemsBase\Dx8Dp\src\core_systems\continuous_time_ode.jl:79 [inlined]
 [10] demo()
    @ Main.SD c:\Users\hswt136nia\OneDrive\Documents\Rock Dene Cottage\Courses\LEAP\Projects\Anatta\src\Development\SD\SD.jl:13
 [11] top-level scope
    @ REPL[1]:1