Integration with C++ : “runtime” compilation?

Hello all,

we have a large C++ scientific codebase that offers a large variety of formulations to the end user. This is implemented through heavy use of template metaprogramming to keep the code compact while avoiding the costs associated with runtime conditionals and unneeded variables. Since the total combination of available options is now running in the billions, it is not practical to distribute a binary with all precompiled compilations. Instead, the codebase is distributed in source form.

The downside is that the end user has to effectively implement a C++ class following specific semantics to select the specific framework template for their desired formulation. I’ve been looking into Julia as a possible way to provide a friendlier interface, with the idea that the user could have a more accessible scripting interface that would, behind the scene, build the specific framework compilation and run it.

I’ve been looking at the obsolete (if my understanding is correct?) Cxx.jl and its successor CxxWrap.jl, but again if my understanding is correct, the former would have worked just the way I intended, whereas CxxWrap effectively requires building the C++ code separately, which largely defeats the point of having a Julia “front end” for the test cases in the first place.

I could work around this by producing from Julia a C++ source to be compiled and then linked from CxxWrap.jl, but I have a feeling that this isn’t exactly “the smart way” to approach the idea. Being new to the Julia ecosystem, I’m quite sure I may be missing something, so any suggestions on how to tackle the problem are welcome.

4 Likes

Hi @Oblomov !

I have been developing a huge satellite simulator for the attitude and orbit control subsystem of a satellite. This simulator contains the C++ version of the firmware so that we can test everything at the software level.

After a lot of thinking and testing, CxxWrap.jl is your best solution. Yes, you need to write glue code in C++ to make the interface with Julia, but it is straightforward. In my case, I use Eigen with a lot of metaprogramming and it works flawlessly.

3 Likes

The glue code is not the problem. The requirement that the class template instances have to be precompiled is. From the CxxWrap.jl documentation:

The natural Julia equivalent of a C++ template class is the parametric type. The mapping is complicated by the fact that all possible parameter values must be compiled in advance,

In my case, the class template that holds the computational framework has billions of possible combinations of template arguments. It’s simply not feasible to precompile all of them.

1 Like

CxxWrap.jl is only a partial successor to Cxx.jl

There are some unrealized plans to provide a successor for the capability of JIT compiling C++ code using GitHub - Gnimuc/ClangCompiler.jl: Clang compiler infrastructure for Julia @Gnimuc may be best be able to discuss the state of things.

I got it to work for some simple examples a while back, but I was lacking the use-case to push it to something like what you are envisioning.

I think one would need to lift some of the code in Cxx.jl that provide the user-interface and re-implement it on top of ClangCompiler.jl

2 Likes

Oh, that looks indeed much more like what I’m looking for. I mean, if all you need is a use-case, I can definitely provide one, and far from a trivial one. Clang would be a good fit: our codebase already supports it as a compiler, and being able to build directly for CUDA or HIP from there would be a nice add-on.

and time… If you have an enthusiastic student who knows C++ a bit it might make a good project.

I see that @Gnimuc has recently added some code for templates, but I don’t know what the status is for a more user-friendly frontend

1 Like

I guess I’ll put this project on hold at the moment, since I too am lacking a bit in time and students :wink: But I’m keeping an eye on the ClangCompiler.jl project, it’s definitely moving in the right direction for my use-case.

Thanks a lot to all.