I’m looking to use Julia for communicating wirelessly with ESP32 boards with a user-friendly and performant API. I felt it would be nice to make a thread on this topic to document the progress which I believe would be helpful to others on similar path with same/different hardware.
This would involve
Compilation of C libraries into jll (julia linked library)
Developing a library that has user friendly julia functions that ccalls the C libraries.
The library would ideally have
Easy methods to create contexts, open connections etc., and also a generic way to support any user defined data type for transfer (with support for in place operations). Mostly following either functional programming paradigm or in place mutations for performance and stability.
Overloaded display functions that pretty print these structures.
Optionally after these are done, routines for pre compilation to speed up package loading, and optimizing the code to make it easier for libraries built on top of these to be static compiled can be targeted. Another potential future target is to abstract away queues, contexts etc. (A very futuristic goal would be to be able to dynamically generate compiled C/assembly code for the embedded device itself and send as a message).
I am currently looking to use ESP Now protocol (open to other protocols as well if they are low overhead on the device side and have good reliability). I’m curious if compiled jll exist for this library already or if I have to compile one. If the latter, I’m looking for guidance on the same since I have not compiled any jll before. My current platform is Windows 11, but I can also use WSL if that’s easier. Given a jll, I have some prior experience in building a user-friendly Julia library!
You might want to look through the Julia artifact registry, GitHub - JuliaPackaging/Yggdrasil: Collection of builder repositories for BinaryBuilder.jl, to see if any of the JLLs you need are there. If they are not there, I think it would be safe to assume that particular JLL does not exist although I might ask @ giordano (left the space to not ping Mose intentionally here) to be sure…
Accelerating Local Development
You can utilize RepliBuild.jl to generate the necessary FFI wrappers before you create the final, multi-platform JLL.
Local Binary Sufficiency: A Julia Linked Library (JLL) is required for reliable package distribution across different operating systems. However, for local development, you only need a single shared library (.dll on Windows or .so on Linux/WSL) compiled specifically for your current machine.
FFI Generation via Introspection: If you can compile the ESP-NOW C/C++ source into a shared library on your local system (for example, within WSL), RepliBuild.jl can use its LLVM/DWARF introspection features to analyze that local binary and its header files.
Automatic Wrapper Creation: This automated process generates the complete Julia module with:
All necessary mutable struct definitions that perfectly match the C++ memory layout.
All function wrappers containing the correct ccall arguments and return types.
This capability effectively eliminates the most tedious and error-prone part of package development—manually transcribing FFI interfaces—allowing you to immediately focus on building the high-level, user-friendly Julia API (contexts, data type handling, display functions).
By using this approach, you create a fully functional, locally-working Julia package first. Once the API is stable, you can then focus your effort on the highly complex task of creating the multi-platform JLL using BinaryBuilder.jl for distribution. Dev the github repo as i fixed enums, arrays, and typedef
Thank you so much. This sounds very cool. I’ll give it a try soon.
Also, I realized that ESP Now library is only available for ESP32 devices. It is possible to use it on non ESP32 devices if we can directly send layer 2 packets. This is something that ESPythoNOW does by leveraging the Scapy Python library. The code itself is pretty small with the heavy lifting being carried out by Scapy.
I believe it would be useful to have a library like Scapy in Julia, if it does not exist yet. I see a jll already for libpcap, but unable to find one for npcap (the latest windows port) and libnet. Or it could be possible to build one on top of the sockets in Julia already.
To implement ESP-NOW on a non-ESP32 device, you need two core capabilities:
Packet Capture (Sniffing): Needed to observe incoming ESP-NOW frames. This is handled by libpcap (or its Windows equivalent, npcap).
Packet Crafting (Injection): Needed to construct and send the custom ESP-NOW frame headers and payloads. This is typically handled by libnet.
While a libpcap_jll exists, a native, comprehensive Julia equivalent to Scapy (a high-level packet manipulation DSL) does not currently have the same maturity. Therefore, the most robust path is to generate Julia FFI wrappers for the lower-level C libraries.
Thank you. I’ll try to implement the FFI, and then the Julia native library for libpcap and libnet then. After that a library like Scapy would be an integration of these two along with a performant and configurable parser library like Construct, nom, binrw, etc