Acausal modeling and Unreal Engine

Specific Domains Modelling & Simulations
, ,
1

I would like to simulate a quadrocopter using Julia and visualize it in Unreal Engine.

I figure ModelingToolkit.jl should let me model the energy consumption of the on-board drone battery and optimize trajectories, etc. to save power. I also want to model PID controllers and the like.

Game engines process physics, etc. every frame, e.g. 30-60 times per second. I understand that acausal modeling is better than causal but can acausal models let me sample the current state of the model every frame?

2

We distinguish between modeling and simulation. Before we simulate an acausal model, the modeling tool typically causalizes it such that it can be passed to a time-stepping ODE/DAE solver for simulation.

If the goal is to visualize only, you wouldn’t even have to worry about the real-time aspects, you can simply pre-compute the trajectory and then generate the visualization afterwards?

1 Like
3

I want to develop the “flight controller” in Julia, deploy it as software in the loop (SITL) in Unreal Engine and make sure the drone behaves realistically. Think rewriting AirSim in Julia.

Once the above is working, I would deploy the controller as hardware in the loop, running on some single-board computer and still attached to Unreal Engine. Finally, I would transfer the flight controller to a drone and test it in the field.

I’m aware of real-time and TTFP issues with Julia. I’m probably not going to use Julia for the flight controller itself but use it for computer vision, navigation, etc.

I’m trying to figure out whether causal or acausal modeling is the way to go in my scenario.

Thanks, Joel
4

You seem to be confused between what it means to model something and what it means to execute or simulate something. Whether or not you use acausal modeling is orthogonal to the concerns you are lifting. The only concern you might have with acausal modeling is whether the particular tool you use (MTK) has support for things like

  • interfacing the software you need for SIL
  • Interfacing the hardware you need for HIL
  • running at a fixed and sufficiently fast sample rate

alternatively, whether or not the tool allows you to export your controller in a format that allows you to call it as a function of state and input.

Currently, MTK has some basic support for calling user-defined functions that could be used for SIL/HIL, but no support for doing this at a fixed sample rate (this PR adds that). MTK is also not very well equipped to export dynamical systems as input-output systems for you to call in your own real-time loop. You can do this to some extent, but it’s rather hacky at the moment.

MTK isn’t a tool for things like computer vision and navigation

1 Like
5

I know what questions to ask now, thanks Fredrik!

The game engine loop is separate from the Julia simulation loop. The latter should update things like the drone position and orientation in space, speed, etc. The game engine will read these values each frame and re-position the drone accordingly.

What do you mean by you can do this to some extent?

Can MTK be hacked to do what I’m looking for?

Should I use MTK for this or is there another Julia package that would work better?

6

If you let the game engine drive the loop, you could easily implement a julia function that the game engine calls each sample instant to retrieve the state of the quadrotor. This function would be stateful, and advance the state of the quadrotor dynamics model a fixed step and output the new state. This would not need any hacks

7

I suspect Julia won’t recalculate the model fast enough to avoid affecting the frame rate of the simulation but…

Can’t I advance the state of the model and output a new state in a separate loop? I can control the frequency of this loop and have it update internal game engine state that will be used to display the quadrocopter.

Can you point me to any resources that will help me implement such a function? What will I use to implement the quadrocopter dynamics model in this case? Can I use MTK and acausal modeling?

8

60Hz for a quadrotor model? I’d be disappointed if you couldn’t update that in at least 10kHz. This is not going to be your problem.

Yes you can, MTK doesn’t force you to model in any particular way, you can model both causally and acausally in MTK.

Here’s an example where MTK is used to model a quadrotor. Unfortunately, the library isn’t released yet so it cannot be used, but it’s nevertheless an indication that it’s possible

1 Like
9

I benchmarked an 18-dimensional quadrotor model integrated one step with RK4, it took

julia> @btime $fdisc($x, $u, 0, 0);
  66.252 ns (0 allocations: 0 bytes)

which would translate to a possible refresh rate of ~15MHz, i.e., as long as your frame rate is less than 10 million frames per second, the performance of simulating a quadrotor in Julia should not be your primary concern :slight_smile:

1 Like
10

I still have a lot to learn, unfortunately…

How do you come up with 18 dimensions for a quadrocopter?

How many steps should you integrate?

Why use RK4?

I did download a bunch of books and papers and plan to dig into them!

11

It depends on your control interval. If this interval is short (which it typically is for a quadrotor), a single step is probably plenty sufficient.

Because it’s simple to implement and offers a reasonable trade off between performance and accuracy, especially when the step size is dictated by control-performance considerations rather than integration accuracy.

I didn’t :stuck_out_tongue: I just took a model I had access to. It has an 18 dimensional state vector, including orientation represented as a rotation matrix.

12

Is seem you want to embed:

and that you’ve chosen Unreal or at least PhysX. I’ve not seen anyone embed it/use with Julia or other game or physics engines I know of (and mention) below except for Bullet. There are some Julia-built game engines (but not physics engines?), and possibly Makie.jl would work for you.

[Since embedding discussed there, I suppose possible, but I thought game engines to be a framework, “don’t call-us-we’ll call you”, so actually you would rather embed Julia in Unreal? And that strictly mean there, using together that way?]

By far the best game engines are, I believe, Unreal, Unity; and for open-source Godot. Historically (and at least somewhat still) the physics simplified in game [engines], and there were separate physics engines, Nvidia’s PhysX (that you may intent to use) PhysX - Wikipedia

PhysX in video games
PhysX technology is used by game engines such as Unreal Engine (version 3 onwards), Unity, Gamebryo, Vision (version 6 onwards), Instinct Engine, Panda3D, Diesel, Torque, HeroEngine, and BigWorld.
[…]

PhysX in other software
[…]

  • Unreal Engine game development software by Epic Games. Unreal Engine 4.26 and onwards has officially deprecated PhysX.
  • Unity by Unity ApS. Unity’s Data-Oriented Technology Stack does not use PhysX.

I’m not sure PhysX or Unreal or any of the game engines can simulate a quadcopter (I suppose a typo or synonym: quadrocopter). I could be wrong. The blades move very fast, and you mean to simulate them, at correct speed, or just visualize, emulate, show illusion of fast moving, e.g. using motion blur or known blade visualization tricks? [In games [helicopter] blades usually don’t move, just look that way, since fully realistic not needed.]

[Maybe remove the games label? Since you’re not making one. If you get any game engine to work with Julia please post about it, then with the label.]

13

I strictly and specifically want to embed Julia into Unreal Engine.

The physics engine built into Unreal does not apply here as all the physics will be done in Julia.

The end result is a game of sorts. It will show a drone flying in various in-game environments (forests, fields, etc.) and driven by physics calculated in Julia. There’s more to it but that’s all I can disclose for now.

2 Likes
14

You can do that, add C++ code (and thus Julia) to Unreal, and (I assume to) get it called somehow, not sure where from, i.e. how, where to plug it in, you do it exactly.

No other language is officially supported, but that doesn’t mean you can call any language from C++. C is close enough, an (almost strict, though not quite) subset of C++, so you can call Julia’s C embedding API, though more likely you would want to use Jluna:

E.g. C#/:net is not officially supported, still people have done this, somehting you would learn from:

Your project will be large because of a) Unreal, but also b) Julia (is large, can be stripped down), you can compile the Julia code and use still large Julia runtime, with PackageCompiler.jl, or if you’re lucky could use StaticCompiler.jl then the Julia part could be a rather small library.

Since you’re only after the visualization part, then maybe you might as well go with Makie.jl, i.e. GLMakie (one game has been made with it… and it’s at least very good as a non-game engine). It should be fast enough. Unreal might be overkill if you just want visualisation, or to start with if you just want a demo, unless you really need and want awesome graphics. There are pretty good game engines like Godot, free/no-cost, that are pretty good to also explore, and free Julia game engines, e.g. B+ most recently in 3D, not yet registered.

About other Julia game engines, e.g. Starlight.jl (current best for 3D?), see my longer post on it:

15

Thank you Páll!

This thread, and my question, is not about embedding Julia in Unreal Engine but rather about getting hold of the internal state of the model during simulation so that I can update game engine state.

It’s also about whether I can use causal or acausal models for my quadrocopter. baggepinnen explained that either one can be used for modeling since

modeling tool typically causalizes it such that it can be passed to a time-stepping ODE/DAE solver for simulation

baggepinnen also pointed out that it should be possible to get hold of the model state during simulation although I still don’t understand how to implement this exactly.

16

You can either

  • Treat each time step as an independent simulation. In this case, you get the model state from the solution object that is returned by solve
  • Implement a PeriodicCallback using DiffEqCallbacks, in which case you get the integrator state from the integrator object that is passed to your callback function.
17

Maybe you should first implement in your simulator a server which can interact with any language through a port.

Try to be able to send commands to this simulator as if it was a “remote control” and maybe after this… you can tackle the other parts of your project (about modeling).

You can probably also consider solutions such as MQTT for that purpose. You will find nearly ready to use solution on both side (ie on simulator side and on Julia side). This solution will decouple your controller from your simulator (but can lead to some latency)