Anyone using Julia for drones (or other embedded)? Any forseen problems?

General Usage
1

[I do know about the general garbage collection issue, that I’m not worried about; solved for robotics, see case studies at Julia Computing.]

Any hardware/OS combination a problem or has been proven to work, and I mostly have this one in mind (seems this CPU/RAM/flash/eMMC capacity is sufficient and Julia would at least work on it with Linux):

https://www.intel.com/content/dam/support/us/en/documents/drones/development-drones/intel-aero-compute-board-guide.pdf

In addition to several fixed-function interfaces […] exposes 6 processor GPIOs, 28 FPGA GPIOs, and 5 FGPA analog sense inputs (ADC). […]

Intel®Atom™ x7-Z8750 Processor 4 cores/4 threads 2.56GHz burst 2M Cache 64-bit
[…]
The HSUART can be accessed through Linux sysfs at node /dev/ttyS1.

seems also to include “Altera® MAX®10 FPGA” (see schematic on page 6). I know Julia doesn’t “support” FPGAs, while not ruling out interfacing to it (or compiling to it in the future; feel free to include info on FPGAs with Julia in any context). It seems this hardware is ideal, with all of it suppored, and I suppose no flags such as this one needed:

julia --cpu-target <target> Limit usage of cpu features up to <target>

https://www.intel.com/content/www/us/en/products/drones/aero-ready-to-fly.html

https://www.intel.com/content/www/us/en/support/articles/000023272/drones/development-drones.html

I also expect all peripherals to work (assuming using Linux) or outside of the scope of Julia, but not Julia with C/C++ libraries/drivers, e.g.: “Intel® RealSense™”.

Do you know if most use just regular Linux, or Linux with a real-time patch (or other RTOS) for drones? I assume Julia would handle Linux with, just as w/o the patch. I.e. Linux API/ABI unchanged.

There’s also e.g. this hardware (with memory/2MB flash assumed too low?) and anyone tried running on this OS or other RTOS:

https://docs.px4.io/en/flight_controller/pixhawk4.html

runs PX4 on the NuttX OS […]
Main FMU Processor: STM32F765

32 Bit Arm® Cortex®-M7, 216MHz, 2MB memory, 512KB RAM

IO Processor: STM32F100

32 Bit Arm® Cortex®-M3, 24MHz, 8KB SRAM

API and library for PX4 Autopilot using MAVLink written in C++11

[I expect all (relevant) C++/C libraries to work with Julia. I recall CxxWrap supports C++14 and higher; i.e. here you would just need to compile C++11 code with a C++14 (or higher) setting.]

https://www.dronecode.org/platform/

In your view would using Julia (and its libraries for e.g. vision and robotics), or Julia+C++ be an improvement over just using what most use (C++). It seems the same argument applies just as for robotics (I’m not sure have many of the Julia libraries in that space apply here).

[Android is getting popular as an embedded OS. Maybe not on the drones themselves, so I don’t care too much for now. I still like to see Android supported…]

FYI: I looked up and found no relevant hits on this Discourse, only on unrelated “drone.io” (here) and there:

2

Well, PX4 cannot run Julia because of lack of supported operating system and memory. Julia runs well on Linux. I would suggest to use a low latency kernel (not RT) and use cpu partitioning to reserve 3 cores for the realtime tasks. You probably can achieve a main loop frequency of 20 Hz with Julia (not sure, you need to test it with your code), but not the 200 to 1000 Hz needed for IMU estimator code. But this might not be needed if you use for example XSense MTI 1 as IMU, as it has its own CPU. And 20 Hz for Drone control is probably also too slow.

The question is, what do you want to use Julia for? It is most likely too slow (too much latency and jitter, not throughput) for the inner control loops. For high level control it might be a good choice. But if you do not know how to program in C you will have a problem. Julia doesn’t solve the two-language problem for realtime flight control. And, in contrast to Matlab/Simulink there is no C code generator that is suitable to create fast, low latency controllers and estimators from high-level definitions.

1 Like
3

You should easily be able to hit 1000 Hz for an EKF-based state estimator, as long as you don’t allocate, as you’ve noted from our case study, since

julia> using BenchmarkTools

julia> @btime GC.gc()
  46.261 ms (0 allocations: 0 bytes)

Over at QPControl.jl, @rdeits and I implemented a basic QP-based controller for a humanoid robot that doesn’t allocate and runs at about 1000 Hz. That controller performs various rigid body dynamics calculations, sets up a QP, and solves it within that millisecond, so I would definitely not expect any problems in terms of throughput for control and state estimation for a drone.

Despite the lack of allocation, we did observe somewhat significant jitter on a Linux machine without special precautions taken. Pegging the CPU frequency helps a bit: using a simple BenchmarkTools test, I’m getting times between 945 μs and 1385 μs right now. We haven’t had the chance to try this on a machine with the realtime kernel patch.

Avoiding allocations can be a hassle, and I’m hoping we’ll soon get new tools to help with this (:wink: @jeff.bezanson).

6 Likes
4

Just as reference: On a raspberry pi 3b+ I can achieve a worst case latency of 60 microseconds with a low latency kernel and cpu shielding. Have a look:
https://stackoverflow.com/questions/11111852/how-to-shield-a-cpu-from-the-linux-scheduler-prevent-it-scheduling-threads-onto

Compiling an RT kernel costs quite some time, is difficult to update and reduces the throughput.

1 Like
5

To mention one more problem: The UDP implementation of Julia is also not very real-time friendly. So if you use UDP to communicate with your sensors and/or actuators you might also come into trouble.

6

@rdeits and I put a lot of work into making GitHub - JuliaRobotics/LCMCore.jl: Low-level Julia bindings for the LCM communications library efficient and making it not allocate (unless you’re sending/receiving strings). LCM is a message passing protocol based on multicast UDP. GitHub - tkoolen/HumanoidLCMSim.jl uses LCMCore.jl to implement a simulation where the controller and simulator run in separate Julia processes that send state information and control actions (from QPConrol.jl) to each other over LCM, still with zero allocation.

7

But that is exactly what I said before: You cannot build a real-time flight control system just with Julia (not at the current point in time). You need to include some C libraries or components. And most likely you need to write ore modify some of them yourself, so you will need some knowledge of C to make the system work.

8

Julia wouldn’t be my top choice for code where you can’t afford jitter. In addition to avoiding allocations, you’d probably also need to avoid all JIT compilation. No doubt it can be done, but IMO with the effort and risk involved I would rather do it in C/C++.

9

Well, avoiding JIT compilation is usually not a big deal. Just make sure all code paths are executed once before you launch your drone. Can easily be done in a proper init function.

10

For smaller pieces of code (e.g. shortish inner loop), or “spaghetti” type of code that’s not structured into a bunch of functions, it might not be a big deal. In general, ensuring that every single function has been called (including “exceptional” code paths not normally run during inner loop), with every set of argument types that it will be called with, does not seem trivial to me.

You would also need to own/control every line of code run. If you depend on another package that’s updated independently, I don’t see how you could ensure that all of its internal functions have been executed?

11

You don’t need to run all the code paths! Julia doesn’t do Java-style HotSpot JIT compilation; it’s much closer to AOT in that regard. You just need to make sure that the ‘main’ function is called once with all the argument types it’ll ever be called with, and that you don’t do any dynamic dispatch (which already follows from the zero allocation requirement).

4 Likes
12

Just a simple example to show this:

@noinline foo(x) = 3 * x
@noinline bar(x) = sin(x) / 4

function baz(x)
    if x > 4
        foo(x)
    else
        bar(x)
    end
end

baz(2.0) # doesn't hit the `x > 4` branch
@allocated foo(5.0)

returns 0. It would certainly not if it needed to compile bar at that point.

EDIT: sorry, this is not a good way to show this property. Getting rid of the baz(2.0) still makes @allocated foo(5.0) return 0.

13

Hmm, ok, thanks for clarifying. I must have misunderstood then. I didn’t think this could be relied upon with recursive function calls, even with static dispatch, cf. this topic.

14

Hi, I agree with @tkoolen that Julia works well for robotics related tasks, and want to emphasize that Julia and C interoperate really well – so no issue there. We use C all the time over by JuliaRobotics, however, we find ourselves moving code into Julia more and more for convenience and speed. Julia itself is blazingly fast, so once the work is scheduled and running there is little difference between C and Julia execution. Maybe the best way to compare is imagine compiling your Python-like code with GCC or Clang … that’s what Julia automatically does for you (using Clang).

There are two additional things I should add here: real-time thread scheduling is an entire discipline on its own and (besides the JIT and GC work-arounds) have almost nothing to do with Julia. Be sure on the difference between hard and soft real-time systems, not to be confused with JIT or GC delays present in first draft (hacky) code. We have some experience with Xenomai which patches the Linux kernel with a hard-real time scheduler and allows you to schedule and interrupt code execution much more accurately than regular Linux. Vanilla Linux is optimized for human usage, and thereby the scheduler allows for jitter times as large is 20-25ms, which allows for larger chunks of thread work at each allocation. That means Vanilla Linux might only service an interrupt 20ms or so later. Hard real-time scheduling performance under load is usually somewhere between 500ns-3us.

One thing the community hasn’t really done is to run Julia code with a different RTOS thread scheduler. You will likely find no difference between precompiled Julia with zero allocations and a C function – other than Julia having many major benefits over just C for technical/numerical/algorithmic computing situations.

The second point is that of static compilation (vs JIT). Yes, tighter control of the compile process would be nice (to look similar to C) but it is not really required in most situations. For this, see ongoing progress at PackageCompiler.jl which can perform static compiling ahead of time. Also remember that Julia code can be compiled to .o, .so, or stand-alone elf files – see some of the Ahead of Time (AOT) compiler docs here. What we find is that once the code is figured out, there is a bit of a cold start delay while code is compiling, but nothing serious – and is lessened by the fact that we spend most our time in development anyway, where compiling remains necessary. In addition, most developers are working to get our Julia packages ready for precompile usage, which means it really only calls the JIT once for fresh source code, regardless of machine reboots thereafter.

Lots of info, sorry, but hope it helps!

6 Likes
15

@blegat, I saw you just liked this post. I just wanted to point to Is Julia the right tool for the job? - #4 by yuyichao where Yichao pointed out that GC.gc() does a full sweep, and a lightly allocating program isn’t likely to trigger a full sweep. No guarantees though.

1 Like
16

Yes, defined:

gc(full::Bool=true)

You can also call GC.gc(false), but you can also fully avoid the GC (being implicitly run), both full sweeps or partial; likely for all code you’re likely to need to avoid the GC for (you need to be careful and make sure you do not allocate). This is one of my favorite Julia case studies, for hard-real-time:

https://juliacomputing.com/case-studies/mit-robotics.html

17

Glad you liked it (@rdeits and I are the authors of that piece). See also our recent paper at https://www.researchgate.net/profile/Twan_Koolen/publication/331983442_Julia_for_robotics_simulation_and_real-time_control_in_a_high-level_programming_language/links/5c98f750a6fdccd4603ae3a2/Julia-for-robotics-simulation-and-real-time-control-in-a-high-level-programming-language.pdf

1 Like
18

I can confirm that Julia works on the Jetson Nano.
For flight use the Nano itself is extremely compact

You can develop on the Jetson Nano Dev kit

You can use Raspberry PI cameras (or compatible) and Raspberry Pi ‘hats’ - these are pin compatible.
You can power the Dev kit from a battery power source.

19

I really would go with the Jetson Nano. There is a community of users out there, and you get a flash image which will run a robot as an alternative to the Ubunbtu install complete with deep learning toolkits etc.
And as I say compatibility with cheap Raspberry Pi sensors and interfaces.