Is there a C/C++/Rust/Go library that can do inference on Flux models?

Specific Domains Machine Learning
, ,
1

If not, is it useful to have one at least for simple layers like Dense, Conv, Scale, etc.?

2

Would building a C lib with PackageCompiler work?

3

Thanks. That looks interesting. Exploring.

4

Where can I find documentation for jl_ functions and types used in C? Thanks.

5

https://docs.julialang.org/en/v1/manual/embedding/

https://docs.julialang.org/en/v1/base/c/

6

Thank you. I did go through those sections. There are very useful examples of function usage, not really documentation. I guess I need to go through the header files.

I got a minimal example with Flux working though.

7

It seems like ONNX.jl is in the middle of some restructuring and is not currently supporting Flux.jl, but otherwise that could possibly be one way of moving a model to another language. Not sure what the exact status is, but does not seem to have too much activity recently :confused:

8

Thank you.

9

That just means you can’t import ONNX files as Flux models (check out GitHub - DrChainsaw/ONNXNaiveNASflux.jl: Import/export ONNX models if you must have that). Exporting to ONNX should work now for a decent number of models. I think the README could use some updating to reflect that, PRs welcome.

2 Likes
10

Thanks. If this can work with C/C++, it would be great.
The C-API route of running a jl environment is not ideal.

11

Once you’ve exported a model to ONNX, you can load it in the runtime of your choice. The most popular one is https://onnxruntime.ai/, which has C/C++ bindings. I know at least a few other runtimes should as well, but not exact names. ONNX | Supported Tools may be a good starting point to read further there.

3 Likes
12

Thank you.