RustCall.jl?

Tooling
1

@ExpandingMan made the point here that calling out to Rust would be a good way to increase the robustness of the Julia ecosystem.

The labor of wrapping a bunch of Rust crates compiled to the different platforms isn’t going to materialize out of nowhere, so my proposal is that tooling to make calling Rust easy for normal users would be very high leverage.

I see that there is a jlrs Rust crate for calling Julia from Rust. Are there technical reasons we don’t have RustCall.jl that installs Cargo and compiles a custom lib.rs with the functions you want to call?

13 Likes
2

It looks like BinaryBuilder.jl can build Rust crates (some docs here and an example here). Is that sufficient for what you’re thinking of?

3 Likes
3

I guess my thought was something like:

using RustCall
# install the crate you need
RustCall.add("some-crate")

Then a block of code to construct a lib.rs with “no mangle” wrapper for the function you want. The wrapper does the necessary for handling the input types (unwrapping pointers and such).

@use begin
    some_crate::useful_function(::Cstring)::Cvoid
end

It would also create a Julia function of the same name which is a wrapper for a @ccall to the Rust function.

It would be a little more pedantic than PythonCall, but so it goes with Rust :sweat_smile:

3 Likes
4

There are some problems that can realistically be solved and some that can’t.

The biggest problem that can’t be solved is that it will always be necessary to write a rust foreign function interface (FFI). In my experience as a rust novice, this is orders of magnitude more difficult than writing “normal” rust code as it essentially means coercing the language into breaking all the safety guarantees it’s built around. Related to this is that there seem to be a number of packages which are maybe half-heartedly intended to be wrapped in other languages, but in practice are only wrapped in python and rely on interpreter shenanigans and the extremely mature and extensive PyO3 crate. As far as I can tell, packages using PyO3 don’t do you many favors if you want to then wrap it for another language. The prominent example here is of course polars.

What I think we can have is something a little bit more like PyO3 for Julia, even if the fact that Julia is compiled as opposed to interpreted means that it may never be as simple. I think even something that was more about workflow and standardization than about actually writing the FFI would be helpful. Right now wrapping a rust project means writing an FFI probably from scratch, and writing the ccall wrapper from scratch. A package that helps developers write the functions to be called on the rust side, and also helps write the C wrapper on the Julia side may be very helpful.

@Taaitaaiger has already done some great work with jlrs. I don’t think I’ve bumped into this person around community forums before, but it would be appreciated if they have some comments to add. It seems likely they are among the most knowledgeable people in this topic. However (and correct me of I’m wrong) it seemed like the focus of that project was more on calling Julia from rust.

When I last seriously looked into this, I was mostly interested in writing a “low level” wrapper of polars that provides views directly into arrow buffers that are allocated on the rust side, and allows rust functions to operate on buffers allocated on the Julia side. At that time, I considered this quite challenging, in no small part because, for a novice rust user like myself it is much, much harder to write an FFI than to just write a rust crate for rust… experienced rust users would surely find it much easier, but I expect rust FFI people are rather specialized and that there are a good number of experienced rust users who would find it quite laborious to write an FFI. Also jlrs didn’t have much to help that particular case, but I know some more work was done on it since then that I have not carefully looked into.

14 Likes
5

It has julia_module which generates the ffi functions if the types implement a trait, but this requires you to create a wrapper crate that and then create a Julia package that references the compiled library file. It might have been clear because it is mixed with a lot of documentation on calling Julia from rust.

I think a goal could be to implement the ability to pass a data frame as arrow back and forth similar to how it’s done in extendr in R, which works quite well. Then you can write your own functions in rust to process tabular data using any crate you want.

6

I wrote this as an example, but it sounds like you already know how to do that.

If there were some additional support, I think it would end up looking more like CxxWrap.jl.

The existence of cargo does seem like a significant difference though, so perhaps a Julia wrapper of some sort would make sense similar to Conda.jl.

For a Julia package though, you probably would just want to use BinaryBuilder.jl to build the the C FFI.

4 Likes
7

I did a very similar proof of concept to your simple example there yesterday! I wish I had found yours first! However, I got tripped up at the very next step – trying to do hello world. Passing a string runs into some of the problems that @ExpandingMan mentioned.

Looking at the docs for BinaryBuilder.jl I don’t see any mention of FFI. Does it do something on the right side to manage passing pointers to Rust?

8

I as outlined in the prior post you need to create a bunch of extern functions from Rust to be able to call things from Julia.

Rust and Julia both understand the C ABI, which is basically the only ABI that is standard on plarforms.

So what I would do on the Rust side, similar to CxxWrap.jl, is to try to automate the creation of the extern functions. It is then this that woild build with BinaryBuilder as Dilum wrote above.

I will get back to you about moving strings between Julia and Rust. The complications there are mainly on the Rust side.

1 Like
9

I was able to get it working with a block of unsafe code that I snagged from a tutorial I found on the internet. But I don’t know enough about Rust to feel confident about automating the generation of unsafe blocks :sweat_smile:

10

As far as Rust is concerned any pointer coming from the outside is potentially unsafe.

11

BinaryBuilder is only a framework for compiling code in some languages into executables or shared libraries. It doesn’t do anything for calling into the shared libraries, apart from providing a variable which points to the library. You build the bindings for a library however you want.

2 Likes
12

I have a package that is an example of writing the FFI from scratch in Rust and then using the ccall interface to call into the Rust code.

I built libcrossterm, which is also built using BinaryBuilder on Yggdrasil, and I used Clang.jl to generate a a Julia interface to that and built a more “Julian” interface on top of that in Crossterm.jl.

Making this process smoother will certainly allow more Rust crates to more easily exposed in Julia.

8 Likes
13

Can Flapigen help? It states that support for any language could be created.

1 Like
14

How do you think we could make this process smoother? Part of the issue here, generating C bindings, seems to be a generic problem for Rust, not one specific to building Julia bindings.

I see that you used cbindgen to generate C bindings. That seems like a first good step to making a Rust binary available to Julia. I see that we do have a Cbindgen_jll.jl although it could use some updating.

The next step would seem to involve reading the generated C header, perhaps using Clang.jl? I wonder about some information loss from this. Perhaps it would be better to talk to something that understands original Rust function signatures such as Cbindgen itself.

Combining these things seems like the basis for a Julia package that can automate the following process:

  1. Use Cbindgen_jll.jl to generate C bindings and headers from Rust code.
  2. Use Clang.jl to generate Julia wrappers to the C code, perhaps with some extra information directly from parsing the Rust code.
  3. Assembling a BinaryBuilder.jl JLL package for the above combination.
  4. Creating a Julia package template that wraps the above JLL.
1 Like
15

I have some unstructured opinions on this topic.

I’ve worked on a number of small projects in this space and as part of them I’ve had various discussions with people about the best way to go about the process of making bindings available in a target language. If I were to generalize, people prefer writing bindings in the language that they are most comfortable with. For interfaces in Python, a C++ developer would prefer pybind11, a C developer might prefer SWIG, a Python developer would prefer cffi + extern C ABI.

To target Julia, a Rust developer would prefer writing pure Rust (the pyo3 equivalent to Julia in Rust). A Julia developer would prefer writing Julia and currently has to write unsafe Rust code.

For a Rust developer, currently jlrs seems like the best way to make a Julia package. First class support for this workflow would help make it easier for those in the Rust community to seamlessly support Julia. This would involve projects that show that this is a viable, templates for getting started, best practices, making it trivially easy to build _jlls, publish them on Yggdrasil, register a package on the official registry etc. (Using jlrs would require making a _jll and then making a package on the General Registry, right? In Rust, if I’m able to build something locally, I can run cargo publish and it’ll become available on crates.io in minutes. cargo allows extensions. Would it be possible to make a cargo julia-publish? Do we need a unofficial alternate registry for crates that are also julia source code + _jlls?)

For a Julia developer, faster iteration of building _jll for the extern C ABI in Rust might help? When I was building Crossterm.jl, if I had to fix a bug that crossed the ABI, I had to make a change to libcrossterm, push a change to Yggdrasil, wait till it was merged, wait till it was registered, then update the dependency locally to test it in Julia. I set up a BinaryBuilder action on GitHub to help but having a faster way to iterate locally would have helped significantly. (If I wanted to use the Rust crate directly, I had to change Julia source code to point to the local shared library and testing this on Windows, MacOS, Linux was a pain. It was easier for me to register it and wait to test it later.)

For a Julia developer, writing unsafe extern C interfaces in Rust and exposing them to Julia also requires dealing with how to handle errors (do you pass in an error object over the C ABI? Set a global variable in Rust? How to translate a panic in Rust to a Julia exception? How to handle errors in a separate thread in Rust?) and ownership (should Rust be the owner or pass back ownership to Julia? what’s the best way to handle multiple references to the same object? Can we use Julia’s GC to clean up Rust objects?). Having guidance on this will make it easier for a motivated Julia developer to dive into Rust and figure out the interconnection.

All that said, I want to say that I’m certainly not suggesting this is a big problem that needs a solution immediately. Python has a much large community of users, and has naturally evolved solutions like swig, pybind11, cffi, pyo3, nimpy etc over time. I think if we can identify low hanging fruit and tackle those problems, we’ll be able to decrease the friction a little. But without a “automagical” solution that makes Rust code “just work” in Julia, I think there’s only so much that we can practically do.

7 Likes
16

Several notes on this point:

  1. If you use the BinaryBuilder.jl package, you can pass the --deploy argument to target your own Github repository. You do not have to Yggdrasil at all. I find Yggdrasil very convenient because I building for all the Julia platforms and microplatforms takes a significant amount of time, but it is not necesary to use. For example, when I am working on HDF5 builds, I will often deploy them to my own Github account before pushing them to Yggdrasil:

  1. Together with the above people can also install packages directly from a git repository or you could setup a new local registry.

  2. You do not need to use BinaryBuilder.jl at all. This is just automation to to help build tarball artifacts and load libraries. You could just build a few tarballs and use the Artifacts system directly to distribute the binaries.

  3. That said, most binaries I build do have other dependencies. I find the practice of using Yggdrasil not just more convenient but also more sustainable since others can help maintain the recipes when other dependencies change.

I think the only current limitation here is a way to trigger JuliaRegistrator from the command line. The path I see there is to use the Github API to create a comment on a commit. The other path is to figure out how the JuliaHub package submission form works.

I do not we think we need another registry for this, but it is possible as I mentioned above. I also do not think we need to use a registry at all but it is convenient.

As I mentioned above you actually do not need to register package on the official registry, Yggdrasil, or use JLLs. You can just push to your Github repository and upload a few tarballs and people could install directly from that. The only slow thing about the Julia registry that I find is the initial registration. I like that the Julia registry seems to do a better job at typo squatting and screening of package names though.

What I’ve not fully accounted for is how good jlrs is. The JlrsCore.jl seems to address much of what I was thinking about already. The issue might be figuring out how initiate this better from the Julia side.

5 Likes
17

This might not be so hard to do actually.

julia> using GitHub

julia> function register_commit(repo, commit_hash, auth)
           GitHub.create_comment(repo, commit_hash, :commit;
               auth,
               params = Dict(:body => "@JuliaRegistrator register")
           )
       end
register_commit (generic function with 1 method)

# Fine Access Token with "contents:read" permissions
julia> myauth = GitHub.authenticate(ENV["GITHUB_AUTH"])
GitHub.OAuth2(github***************************************************************************************)

julia> register_commit("mkitti/ColorVectorSpace.jl", "35ddfef9932583fb4a62ad2021086b2758fbc6e6", myauth)

Comment (all fields are Union{Nothing, T}):
  body: "@JuliaRegistrator register"
  commit_id: "35ddfef9932583fb4a62ad2021086b2758fbc6e6"
  id: 141193778
  created_at: DateTime("2024-04-21T07:16:37")
  updated_at: DateTime("2024-04-21T07:16:37")
  url: URI("https://api.github.com/repos/mkitti/ColorVectorSpace.jl/comments/141193778")
  html_url: URI("https://github.com/mkitti/ColorVectorSpace.jl/commit/35ddfef9932583fb4a62ad2021086b2758fbc6e6#commitcomment-141193778")
  user: Owner("mkitti")
5 Likes
18

Thank you for your kind words about jlrs!

As several people have already commented, the main challenge is that most Rust crates don’t have a C API and the Rust ABI is inherently unstable. This forces you to write a custom C API and target that from Julia. On the Julia side you’re similarly required to provide matching implementations of any non-primitive type, and write functions that ccall the exported functions.

The main advantage of manually writing these bindings is that they won’t depend on Julia. When new versions of Julia is released, you can be confident your library will still work.

The approach I’ve taken in jlrs is strongly inspired by CxxWrap.jl. The julia_module macro converts exported types and functions into one big (re)initialization function which can be called to generate the module in a way that’s friendly to precompilation. The trade-off is that while using jlrs will generate a lot of boilerplate code for you, it requires depending on Julia, which means your library can only support Julia versions supported by jlrs.

The main roadblock for providing such functionality independently of the targeted version of Julia is the lack of a stable C API. At the end of the day, the current C API in julia.h is an internal API that exposes what Julia needs with relatively little care for external consumers. Both CxxWrap.jl and jlrs use this API to be able to expose enough information to Julia to generate the module body. Either those parts have to be stabilized, or some stable interface layer has to be provided by Julia.

10 Likes
19

Just a note, @ZuseZ4 posted this topic (kindly! Thank you!) over to the Rust Reddit and it has gotten a bit of engagement from Rust side: Reddit - Dive into anything

Just wanted to let you all know!

3 Likes
20

I can’t help much with ffi, but would like to see more rust/julia interop.
Long-term, there is some intention on the Rust side to make interop more convenient for languages that don’t want to express all their types in terms of what C has to offer:

6 Likes