Anyone using Julia for drones (or other embedded)? Any forseen problems?


#1

[I do know about the general garbage collection issue, that I’m not worried about; solved for robotics, see case studies at Julia Computing.]

Any hardware/OS combination a problem or has been proven to work, and I mostly have this one in mind (seems this CPU/RAM/flash/eMMC capacity is sufficient and Julia would at least work on it with Linux):

In addition to several fixed-function interfaces […] exposes 6 processor GPIOs, 28 FPGA GPIOs, and 5 FGPA analog sense inputs (ADC). […]

Intel®Atom™ x7-Z8750 Processor 4 cores/4 threads 2.56GHz burst 2M Cache 64-bit
[…]
The HSUART can be accessed through Linux sysfs at node /dev/ttyS1.

seems also to include “Altera® MAX®10 FPGA” (see schematic on page 6). I know Julia doesn’t “support” FPGAs, while not ruling out interfacing to it (or compiling to it in the future; feel free to include info on FPGAs with Julia in any context). It seems this hardware is ideal, with all of it suppored, and I suppose no flags such as this one needed:

julia --cpu-target <target> Limit usage of cpu features up to <target>

I also expect all peripherals to work (assuming using Linux) or outside of the scope of Julia, but not Julia with C/C++ libraries/drivers, e.g.: “Intel® RealSense™”.

Do you know if most use just regular Linux, or Linux with a real-time patch (or other RTOS) for drones? I assume Julia would handle Linux with, just as w/o the patch. I.e. Linux API/ABI unchanged.

There’s also e.g. this hardware (with memory/2MB flash assumed too low?) and anyone tried running on this OS or other RTOS:

https://docs.px4.io/en/flight_controller/pixhawk4.html

runs PX4 on the NuttX OS […]
Main FMU Processor: STM32F765

32 Bit Arm® Cortex®-M7, 216MHz, 2MB memory, 512KB RAM

IO Processor: STM32F100

32 Bit Arm® Cortex®-M3, 24MHz, 8KB SRAM

API and library for PX4 Autopilot using MAVLink written in C++11

[I expect all (relevant) C++/C libraries to work with Julia. I recall CxxWrap supports C++14 and higher; i.e. here you would just need to compile C++11 code with a C++14 (or higher) setting.]

https://www.dronecode.org/platform/

In your view would using Julia (and its libraries for e.g. vision and robotics), or Julia+C++ be an improvement over just using what most use (C++). It seems the same argument applies just as for robotics (I’m not sure have many of the Julia libraries in that space apply here).

[Android is getting popular as an embedded OS. Maybe not on the drones themselves, so I don’t care too much for now. I still like to see Android supported…]

FYI: I looked up and found no relevant hits on this Discourse, only on unrelated “drone.io” (here) and there:


Support wanted/needed for alternative platforms? E.g. Haiku, what's needed?
#2

Well, PX4 cannot run Julia because of lack of supported operating system and memory. Julia runs well on Linux. I would suggest to use a low latency kernel (not RT) and use cpu partitioning to reserve 3 cores for the realtime tasks. You probably can achieve a main loop frequency of 20 Hz with Julia (not sure, you need to test it with your code), but not the 200 to 1000 Hz needed for IMU estimator code. But this might not be needed if you use for example XSense MTI 1 as IMU, as it has its own CPU. And 20 Hz for Drone control is probably also too slow.

The question is, what do you want to use Julia for? It is most likely too slow (too much latency and jitter, not throughput) for the inner control loops. For high level control it might be a good choice. But if you do not know how to program in C you will have a problem. Julia doesn’t solve the two-language problem for realtime flight control. And, in contrast to Matlab/Simulink there is no C code generator that is suitable to create fast, low latency controllers and estimators from high-level definitions.


#3

You should easily be able to hit 1000 Hz for an EKF-based state estimator, as long as you don’t allocate, as you’ve noted from our case study, since

julia> using BenchmarkTools

julia> @btime GC.gc()
  46.261 ms (0 allocations: 0 bytes)

Over at QPControl.jl, @rdeits and I implemented a basic QP-based controller for a humanoid robot that doesn’t allocate and runs at about 1000 Hz. That controller performs various rigid body dynamics calculations, sets up a QP, and solves it within that millisecond, so I would definitely not expect any problems in terms of throughput for control and state estimation for a drone.

Despite the lack of allocation, we did observe somewhat significant jitter on a Linux machine without special precautions taken. Pegging the CPU frequency helps a bit: using a simple BenchmarkTools test, I’m getting times between 945 μs and 1385 μs right now. We haven’t had the chance to try this on a machine with the realtime kernel patch.

Avoiding allocations can be a hassle, and I’m hoping we’ll soon get new tools to help with this (:wink: @jeff.bezanson).


#4

Just as reference: On a raspberry pi 3b+ I can achieve a worst case latency of 60 microseconds with a low latency kernel and cpu shielding. Have a look:

Compiling an RT kernel costs quite some time, is difficult to update and reduces the throughput.


#5

To mention one more problem: The UDP implementation of Julia is also not very real-time friendly. So if you use UDP to communicate with your sensors and/or actuators you might also come into trouble.


#6

@rdeits and I put a lot of work into making https://github.com/JuliaRobotics/LCMCore.jl efficient and making it not allocate (unless you’re sending/receiving strings). LCM is a message passing protocol based on multicast UDP. https://github.com/tkoolen/HumanoidLCMSim.jl uses LCMCore.jl to implement a simulation where the controller and simulator run in separate Julia processes that send state information and control actions (from QPConrol.jl) to each other over LCM, still with zero allocation.


#7

But that is exactly what I said before: You cannot build a real-time flight control system just with Julia (not at the current point in time). You need to include some C libraries or components. And most likely you need to write ore modify some of them yourself, so you will need some knowledge of C to make the system work.


#8

Julia wouldn’t be my top choice for code where you can’t afford jitter. In addition to avoiding allocations, you’d probably also need to avoid all JIT compilation. No doubt it can be done, but IMO with the effort and risk involved I would rather do it in C/C++.


#9

Well, avoiding JIT compilation is usually not a big deal. Just make sure all code paths are executed once before you launch your drone. Can easily be done in a proper init function.


#10

For smaller pieces of code (e.g. shortish inner loop), or “spaghetti” type of code that’s not structured into a bunch of functions, it might not be a big deal. In general, ensuring that every single function has been called (including “exceptional” code paths not normally run during inner loop), with every set of argument types that it will be called with, does not seem trivial to me.

You would also need to own/control every line of code run. If you depend on another package that’s updated independently, I don’t see how you could ensure that all of its internal functions have been executed?


#11

You don’t need to run all the code paths! Julia doesn’t do Java-style HotSpot JIT compilation; it’s much closer to AOT in that regard. You just need to make sure that the ‘main’ function is called once with all the argument types it’ll ever be called with, and that you don’t do any dynamic dispatch (which already follows from the zero allocation requirement).


#12

Just a simple example to show this:

@noinline foo(x) = 3 * x
@noinline bar(x) = sin(x) / 4

function baz(x)
    if x > 4
        foo(x)
    else
        bar(x)
    end
end

baz(2.0) # doesn't hit the `x > 4` branch
@allocated foo(5.0)

returns 0. It would certainly not if it needed to compile bar at that point.

EDIT: sorry, this is not a good way to show this property. Getting rid of the baz(2.0) still makes @allocated foo(5.0) return 0.


#13

Hmm, ok, thanks for clarifying. I must have misunderstood then. I didn’t think this could be relied upon with recursive function calls, even with static dispatch, cf. this topic.