# Acausal modeling and Unreal Engine

**URL:** <https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016>\
**Category:** Modelling & Simulations\
**Tags:** modelingtoolkit, modelling, games\
**Created:** [March 1, 2024, 2:12pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016 "2024-03-01T14:12:40Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 1, 2024, 2:12pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/1 "2024-03-01T14:12:40Z")

</div>

I would like to simulate a quadrocopter using Julia and visualize it in Unreal Engine.

I figure ModelingToolkit.jl should let me model the energy consumption of the on-board drone battery and optimize trajectories, etc. to save power. I also want to model PID controllers and the like.

Game engines process physics, etc. every frame, e.g. 30-60 times per second. I understand that acausal modeling is better than causal but can acausal models let me sample the current state of the model every frame?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 1, 2024, 2:31pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/2 "2024-03-01T14:31:11Z")

</div>

We distinguish between modeling and simulation. Before we simulate an acausal model, the modeling tool typically causalizes it such that it can be passed to a time-stepping ODE/DAE solver for simulation.

If the goal is to visualize only, you wouldn’t even have to worry about the real-time aspects, you can simply pre-compute the trajectory and then generate the visualization afterwards?

---

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 1, 2024, 2:39pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/3 "2024-03-01T14:39:57Z")

</div>

I want to develop the “flight controller” in Julia, deploy it as _software in the loop_ (SITL) in Unreal Engine and make sure the drone behaves realistically. Think rewriting AirSim in Julia.

Once the above is working, I would deploy the controller as _hardware in the loop_, running on some single-board computer and still attached to Unreal Engine. Finally, I would transfer the flight controller to a drone and test it in the field.

I’m aware of real-time and TTFP issues with Julia. I’m probably not going to use Julia for the flight controller itself but use it for computer vision, navigation, etc.

I’m trying to figure out whether causal or acausal modeling is the way to go in my scenario.

```
Thanks, Joel

```

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 1, 2024, 4:00pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/4 "2024-03-01T16:00:57Z")

</div>

You seem to be confused between what it means to model something and what it means to execute or simulate something. Whether or not you use acausal modeling is orthogonal to the concerns you are lifting. The only concern you might have with acausal modeling is whether the _particular tool_ you use (MTK) has support for things like

- interfacing the software you need for SIL
- Interfacing the hardware you need for HIL
- running at a fixed and sufficiently fast sample rate

alternatively, whether or not the tool allows you to export your controller in a format that allows you to call it as a function of state and input.

Currently, MTK has some basic support for calling user-defined functions that could be used for SIL/HIL, but no support for doing this at a fixed sample rate ([this PR adds that](https://github.com/SciML/ModelingToolkit.jl/pull/2414)). MTK is also not very well equipped to export dynamical systems as input-output systems for you to call in your own real-time loop. You can do this to some extent, but it’s rather hacky at the moment.

* * *

> [@joelreymont](#):
>
> I’m probably not going to use Julia for the flight controller itself but use it for computer vision, navigation, etc.

MTK isn’t a tool for things like computer vision and navigation

---

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 1, 2024, 4:18pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/5 "2024-03-01T16:18:09Z")

</div>

I know what questions to ask now, thanks Fredrik!

The game engine loop is separate from the Julia simulation loop. The latter should update things like the drone position and orientation in space, speed, etc. The game engine will read these values each frame and re-position the drone accordingly.

What do you mean by _you can do this to some extent_?

Can MTK be hacked to do what I’m looking for?

Should I use MTK for this or is there another Julia package that would work better?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 1, 2024, 4:31pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/6 "2024-03-01T16:31:49Z")

</div>

If you let the game engine drive the loop, you could easily implement a julia function that the game engine calls each sample instant to retrieve the state of the quadrotor. This function would be stateful, and advance the state of the quadrotor dynamics model a fixed step and output the new state. This would not need any hacks

---

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 1, 2024, 5:13pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/7 "2024-03-01T17:13:29Z")

</div>

I suspect Julia won’t recalculate the model fast enough to avoid affecting the frame rate of the simulation but…

Can’t I advance the state of the model and output a new state in a _separate_ loop? I can control the frequency of this loop and have it update internal game engine state that will be used to display the quadrocopter.

Can you point me to any resources that will help me implement such a function? What will I use to implement the quadrocopter dynamics model in this case? Can I use MTK and acausal modeling?

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 1, 2024, 6:39pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/8 "2024-03-01T18:39:06Z")

</div>

> [@joelreymont](#):
>
> I suspect Julia won’t recalculate the model fast enough to avoid affecting the frame rate of the simulation but…

60Hz for a quadrotor model? I’d be disappointed if you couldn’t update that in _at least_ 10kHz. This is not going to be your problem.

> [@joelreymont](#):
>
> What will I use to implement the quadrocopter dynamics model in this case? Can I use MTK and acausal modeling?

Yes you can, MTK doesn’t force you to model in any particular way, you can model both causally and acausally in MTK.

Here’s an example where MTK is used to model a quadrotor. Unfortunately, the library isn’t released yet so it cannot be used, but it’s nevertheless an indication that it’s possible

> **[Home · AerialVehicles.jl](https://help.juliahub.com/drone/dev/)**
>
> Documentation for AerialVehicles.jl.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 2, 2024, 11:01am UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/9 "2024-03-02T11:01:40Z")

</div>

I benchmarked an 18-dimensional quadrotor model integrated one step with RK4, it took

```julia-repl
julia> @btime $fdisc($x, $u, 0, 0);
  66.252 ns (0 allocations: 0 bytes)

```

which would translate to a possible refresh rate of ~15MHz, i.e., as long as your frame rate is less than 10 million frames per second, the performance of simulating a quadrotor in Julia should not be your primary concern 🙂

---

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 2, 2024, 11:51am UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/10 "2024-03-02T11:51:56Z")

</div>

> [@baggepinnen](#):
>
> I benchmarked an 18-dimensional quadrotor model integrated one step with RK4

I still have a lot to learn, unfortunately…

How do you come up with 18 dimensions for a quadrocopter?

How many steps should you integrate?

Why use RK4?

I did download a bunch of books and papers and plan to dig into them!

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 2, 2024, 12:40pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/11 "2024-03-02T12:40:57Z")

</div>

> [@joelreymont](#):
>
> How many steps should you integrate?

It depends on your control interval. If this interval is short (which it typically is for a quadrotor), a single step is probably plenty sufficient.

> [@joelreymont](#):
>
> Why use RK4?

Because it’s simple to implement and offers a reasonable trade off between performance and accuracy, especially when the step size is dictated by control-performance considerations rather than integration accuracy.

> [@joelreymont](#):
>
> How do you come up with 18 dimensions for a quadrocopter?

I didn’t 😛 I just took a model I had access to. It has an 18 dimensional state vector, including orientation represented as a rotation matrix.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [March 2, 2024, 7:13pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/12 "2024-03-02T19:13:31Z")

</div>

Is seem you want to embed:

> <https://stackoverflow.com/questions/66178438/embed-unreal-engine-4-project-into-another-app>

and that you’ve chosen Unreal or at least PhysX. I’ve not seen anyone embed it/use with Julia or other game or physics engines I know of (and mention) below except for Bullet. There are some Julia-built game engines (but not physics engines?), and possibly Makie.jl would work for you.

[Since embedding discussed there, I suppose possible, but I thought game engines to be a framework, “don’t call-us-we’ll call you”, so actually you would rather embed Julia in Unreal? And that strictly mean there, using together that way?]

By far the best game engines are, I believe, Unreal, Unity; and for open-source Godot. Historically (and at least somewhat still) the physics simplified in game [engines], and there were separate physics engines, Nvidia’s PhysX (that you may intent to use) [PhysX - Wikipedia](https://en.wikipedia.org/wiki/PhysX#Usage)

> **PhysX in video games**  
> PhysX technology is used by game engines such as Unreal Engine (version 3 onwards), Unity, Gamebryo, Vision (version 6 onwards), Instinct Engine, Panda3D, Diesel, Torque, HeroEngine, and BigWorld.  
> […]
> 
> **PhysX in other software**  
> […]
> 
> - Unreal Engine game development software by Epic Games. **Unreal Engine 4.26 and onwards has officially deprecated PhysX.**
> - Unity by Unity ApS. Unity’s Data-Oriented Technology Stack does not use PhysX.

> [@joelreymont](#):
>
> simulate a quadrocopter using Julia and visualize it in Unreal Engine.

I’m not sure PhysX or Unreal or any of the game engines can simulate a quadcopter (I suppose a typo or synonym: quadrocopter). I could be wrong. The blades move very fast, and you mean to simulate them, at correct speed, or just visualize, emulate, show illusion of fast moving, e.g. using motion blur or known blade visualization tricks? [In games [helicopter] blades usually don’t move, just look that way, since fully realistic not needed.]

> **[What are MIL, SIL, PIL, and HIL, and how do they integrate with the...](https://www.mathworks.com/matlabcentral/answers/440277-what-are-mil-sil-pil-and-hil-and-how-do-they-integrate-with-the-model-based-design-approach)**
>
> What are MIL, SIL, PIL, and HIL and how do they integrate with the Model-Based Design approach?

[Maybe remove the games label? Since you’re not making one. If you get any game engine to work with Julia please post about it, then with the label.]

---

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 2, 2024, 7:44pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/13 "2024-03-02T19:44:12Z")

</div>

I strictly and specifically want to embed _Julia_ into Unreal Engine.

The physics engine built into Unreal does not apply here as all the physics will be done in Julia.

The end result is a game of sorts. It will show a drone flying in various in-game environments (forests, fields, etc.) and driven by physics calculated in Julia. There’s more to it but that’s all I can disclose for now.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [March 3, 2024, 4:06pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/14 "2024-03-03T16:06:50Z")

</div>

You can do that, add C++ code (and thus Julia) to Unreal, and (I assume to) get it called somehow, not sure where from, i.e. how, where to plug it in, you do it exactly.

No other language is officially supported, but that doesn’t mean you can call any language from C++. C is close enough, an (almost strict, though not quite) subset of C++, so you can call Julia’s C embedding API, though more likely you would want to use Jluna:

> [@Jluna: a new Julia \<-\> C++ wrapper](https://discourse.julialang.org/t/jluna-a-new-julia-c-wrapper/77486):
>
> Available here: [https://github.com/Clemapfel/jluna](https://github.com/Clemapfel/jluna) (Edit: the license of the library has been changed to MIT, making it freely available without restrictions) I have spend the last few months, full-time, writing a Julia ⭤ wrapper completely from scratch. It aims to fully replace the C-API in usage in C++ projects, and offers a possible alternative to Cxx.jl. I aim for jluna to be… modern: uses the most modern C++20 features, the newest compiler versions and Julia 1.7+ generic: all std:: …

E.g. C#/:net is not officially supported, still people have done this, somehting you would learn from:

> **[GitHub - nxrighthere/UnrealCLR: Unreal Engine .NET 6 integration](https://github.com/nxrighthere/UnrealCLR)**
>
> Unreal Engine .NET 6 integration

> **[C# in UE](https://forums.unrealengine.com/t/c-in-ue/574058)**
>
> i really wanted to start programing in UE, idk which language to use, ik C++ is the default one, my friend said to that C# is without comparison the best one how i use C# in UE? and which is the best?

> **[What Coding Language Does Unreal Engine Use | Robots.net](https://robots.net/tech/what-coding-language-does-unreal-engine-use/)**
>
> Unreal Engine, a powerful game development platform, primarily utilizes C++ as its coding language, offering developers endless possibilities in creating immersive and realistic virtual experiences.

Your project will be large because of a) Unreal, but also b) Julia (is large, can be stripped down), you can compile the Julia code and use still large Julia runtime, with PackageCompiler.jl, or if you’re lucky could use StaticCompiler.jl then the Julia part could be a rather small library.

Since you’re only after the visualization part, then maybe you might as well go with Makie.jl, i.e. GLMakie (one game has been made with it… and it’s at least very good as a non-game engine). It should be fast enough. Unreal might be overkill if you just want visualisation, or to start with if you just want a demo, unless you really need and want awesome graphics. There are pretty good game engines like Godot, free/no-cost, that are pretty good to also explore, and free Julia game engines, e.g. B+ most recently in 3D, not yet registered.

About other Julia game engines, e.g. Starlight.jl (current best for 3D?), see my longer post on it:

> [@Gamedev with GLMakie (the path less taken). Is it possible?](https://discourse.julialang.org/t/gamedev-with-glmakie-the-path-less-taken-is-it-possible/110733/8):
>
> It’s an option, as shown by (the currently marked solution, though I dare say it’s the wrong option, and better available even in Julia for games; for plots, maybe nothing as good in any language), and I see in the docs: Add one or more of the Makie backend packages GLMakie.jl (OpenGL), CairoMakie.jl (Cairo), or WGLMakie.jl (WebGL), RPRMakie (RadeonProRender) [All of those support 3D, even CairoMakie, and RPRMakie ray-tracing. You may want that and OpenGL possibly, or its successor Vulkan.] …

---

<div class="post-metadata">

**Author:** ![joelreymont](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joelreymont/32/206998_2.png) [@joelreymont](https://discourse.julialang.org/u/joelreymont)\
**Post date:** [March 3, 2024, 4:27pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/15 "2024-03-03T16:27:30Z")

</div>

Thank you Páll!

This thread, and my question, is not about embedding Julia in Unreal Engine but rather about getting hold of the _internal state_ of the model during simulation so that I can update game engine state.

It’s also about whether I can use causal or acausal models for my quadrocopter. [baggepinnen](https://discourse.julialang.org/u/baggepinnen) explained that either one can be used for modeling since

> modeling tool typically causalizes it such that it can be passed to a time-stepping ODE/DAE solver for simulation

[baggepinnen](https://discourse.julialang.org/u/baggepinnen) also [pointed out](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/6) that it should be possible to get hold of the model state during simulation although I still don’t understand _how to implement this exactly_.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [March 4, 2024, 5:18am UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/16 "2024-03-04T05:18:37Z")

</div>

> [@joelreymont](#):
>
> [baggepinnen](https://discourse.julialang.org/u/baggepinnen) also [pointed out](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/6) that it should be possible to get hold of the model state during simulation although I still don’t understand _how to implement this exactly_.

You can either

- Treat each time step as an independent simulation. In this case, you get the model state from the solution object that is returned by `solve`
- Implement a `PeriodicCallback` using DiffEqCallbacks, in which case you get the integrator state from the integrator object that is passed to your callback function.

---

<div class="post-metadata">

**Author:** ![scelles](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scelles/32/220408_2.png) [@scelles](https://discourse.julialang.org/u/scelles)\
**Post date:** [May 8, 2024, 9:02pm UTC](https://discourse.julialang.org/t/acausal-modeling-and-unreal-engine/111016/17 "2024-05-08T21:02:37Z")

</div>

Maybe you should first implement in your simulator a server which can interact with any language through a port.

Try to be able to send commands to this simulator as if it was a “remote control” and maybe after this… you can tackle the other parts of your project (about modeling).

You can probably also consider solutions such as MQTT for that purpose. You will find nearly ready to use solution on both side (ie on simulator side and on Julia side). This solution will decouple your controller from your simulator (but can lead to some latency)