# Real-time input and plotting with ModelingToolkit

**URL:** <https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713>\
**Category:** Modelling & Simulations\
**Created:** [September 30, 2024, 8:44am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713 "2024-09-30T08:44:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [September 30, 2024, 8:44am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/1 "2024-09-30T08:44:30Z")

</div>

I’d like to have an MTK model like `D(x) = k*x`, and plot the solution in real time (after 1 second, the plot shows the solution from t=0 to t=1, etc.)

I’d like `k` to be supplied by a real-time interactive slider (so that when the user moves it, the real-time plot reacts accordingly). What’s the easiest way to achieve something like that? Ideally in Pluto (perhaps this is asking too much 🙂)

From [Playing with live plots in Pluto - Specific Domains / Visualization - Julia Programming Language (julialang.org)](https://discourse.julialang.org/t/playing-with-live-plots-in-pluto/44062) I can make pluto-side animations, but each time-step has to be a single call (i.e. that wouldn’t work with a callback). So at each animation frame I’d have to setup the MTK problem and solve it over `(current_t, current_t+1/30)` (to target 30FPS). This might be doable, but I’d have to build the current total solution (t=0 to t=current\_t) from the individual solutions, and set up each ODEProblem’s initial condition from the previous ODEProblem’s final values. This would definitely work, but it looks rather painful to setup.

Perhaps there is a solution using MTK callbacks? As in, I setup a periodic callback corresponding to my target FPS, somehow update `k` from the current slider value, somehow update the plot, then `sleep()` until the next frame? I’m a lot fuzzier on the details of that solution. It might involve… threads?

Somewhat related: [SciML: optimizing a control signal - Specific Domains / Modelling & Simulations - Julia Programming Language (julialang.org)](https://discourse.julialang.org/t/sciml-optimizing-a-control-signal/117504)

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [September 30, 2024, 10:11am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/2 "2024-09-30T10:11:33Z")

</div>

Here’s an implementation of a discrete callback that makes the simulation progress in real time

> <https://github.com/SciML/ModelingToolkit.jl/blob/e8263e9c06e7b0837cadffcbee2c3dc5f22c639a/src/systems/clock_inference.jl#L248C2-L262C73>

you can probably adapt this to your use case. To achieve the slider controlling a parameter, you can probably go through a global variable, but you need to make sure that the simulation and the slider update happens on different threads. The solution I linked above uses `systemsleep` which does not yield, but has better accuracy than `sleep`. If your real-time requirements are not very tight, you can use `sleep` instead, which is yielding. Then it’s enough for the simulation and the slider to run on different tasks (which may be on the same thread).

---

<div class="post-metadata">

**Author:** ![Bart\_van\_de\_Lint](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bart_van_de_lint/32/212161_2.png) [@Bart\_van\_de\_Lint](https://discourse.julialang.org/u/Bart_van_de_Lint)\
**Post date:** [September 30, 2024, 10:40am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/3 "2024-09-30T10:40:25Z")

</div>

@ufechner7 is this possible with ControlPlots.jl?

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [September 30, 2024, 11:27am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/4 "2024-09-30T11:27:10Z")

</div>

Callbacks are one option. Another option is to run the simulation for 0…dt, then update the display, than from dt…2dt etc.

Example code:

```julia
function simulate(integrator, steps)
    for i in 1:steps
        KiteModels.next_step!(kps4, integrator; set_speed=0, dt)

        sys_state = SysState(kps4)
        q = QuatRotation(sys_state.orient)
        roll, pitch, yaw = rad2deg.(quat2euler(q))
        println("roll: ", roll, " pitch: ", pitch, " yaw: ", yaw)
        
        reltime = i*dt-dt
        if mod(i, 5) == 1
            plot2d(kps4.pos, reltime; zoom=ZOOM, xlim=(40,60), front=FRONT_VIEW, segments=set.segments)                       
        end
    end
end

```

In this example I run the simulation with steps of 20ms and update the GUI on every 5th step.

`ControlPlots.jl` does not (yet) offer sliders, for sliders you must use other options like `Makie.jl` or `QML.jl` or others.

The [integrator interface](https://docs.sciml.ai/DiffEqDocs/stable/basics/integrator/) from `DifferentialEquations.jl` can be used with MTK models.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [October 1, 2024, 1:47am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/5 "2024-10-01T01:47:35Z")

</div>

That looks perfect for the Pluto interaction code, I’ll give it a try!

One question: from reading the integrator interface documentation, it looks like mutating the state between each `step!` call is not a good idea. So I guess I should set up a periodic callback to read the slider’s current value and assign it to `k`. I’m assuming that this is OK? Are periodic callbacks ever called more than once for a given `t`? (in which case, they might read a different value, which sounds bad…?)

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [October 1, 2024, 6:25am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/6 "2024-10-01T06:25:45Z")

</div>

> [@cstjean](#):
>
> mutating the state between each `step!` call is not a good idea

Well, normally a system has inputs that are different variables than the state variables. You can always modify the inputs. No need for callbacks.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [October 2, 2024, 9:02am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/7 "2024-10-02T09:02:10Z")

</div>

Worked great, thank you. For the Pluto interaction I had to use a function of `t` which accesses a global `Ref`, that is set by the Pluto slider (through another intermediary variable + cell). So quite roundabout, but nothing too difficult.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [November 1, 2024, 6:59am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/8 "2024-11-01T06:59:22Z")

</div>

> [@ufechner7](#):
>
> Well, normally a system has inputs that are different variables than the state variables. You can always modify the inputs. No need for callbacks.

This was working fine for a while, but after reformulating my model I’m now hitting return codes of `Unstable`. To be clear, I’ve got a loop like

```julia
my_value = 10
read_my_value(t) = my_value
@register_symbolic read_my_value(t)
while true
    step!(integrator, dt, true)
    my_value += 1 # super-simplified; in reality it's a GUI slider
end

```

I read the [PSA](https://discourse.julialang.org/t/psa-how-to-help-yourself-debug-differential-equation-solving-issues/62489) and this part:

> Double check if you’re doing anything that violates assumptions of being an ODE. As an ODE, the right-hand side `f` function should always give you the same result: `u' = f(u,p,t)` needs to be uniquely defined. These issues are just fundamental to the mathematics: if you do these things in `f` , then `f` no longer defines an ODE so of course it cannot be solved!

makes me worry that maybe `step!` can go before the current time `t` and get upset that `f(t-1)` no longer returns the value it did before.

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [November 1, 2024, 8:31am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/9 "2024-11-01T08:31:06Z")

</div>

I’ve basically confirmed from experiment that

- A) The solver does go back in the past, to values before `integrator.t`
- B) Depending on the equations MTK simplifies to, it may not appreciate sudden jumps in inputs (yielding Unstable, or MaxIter)

My solution is to have a `LinearInterpolation`, and `push!` onto its `t` and `u`. That itself may be dangerous, but for now it satisfies the solver…

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [November 1, 2024, 8:37am UTC](https://discourse.julialang.org/t/real-time-input-and-plotting-with-modelingtoolkit/120713/10 "2024-11-01T08:37:00Z")

</div>

> [@cstjean](#):
>
> - A) The solver does go back in the past, to values before `integrator.t`

Yes, it cannot go before `integrator.t`.

> [@cstjean](#):
>
> - B) Depending on the equations MTK simplifies to, it may not appreciate sudden jumps in inputs (yielding Unstable, or MaxIter)

Yes. That’s general for ODE solvers.

> [@cstjean](#):
>
> My solution is to have a `LinearInterpolation`, and `push!` onto its `t` and `u`. That itself may be dangerous, but for now it satisfies the solver…

You might want to `u_modified!(integrator. true)` to let it know that you just placed a derivative discontinuity when using the integrator interface live like that.