Local expr with local variables - aka metaprogramming C++

General Usage
, , ,
1

Hi all,

I am trying to use a C++ library by creating C++ code dynamically (i.e. meta-programming C++). I’m using Cxx package for this.

In Cxx there’s a macro-string named icxx_str which basically wraps the code inside the argument string in a C++ function and executes it.

It looks like this (source code):

macro icxx_str(str, args...)
    annotations = length(args) > 0 ? first(args) : ""
    esc(process_cxx_string(str, false, false, __source__, annotations))
end

A simple usage:

# cxx is for global scope
cxx"include <iostream>;"

# icxx can interpolate variables
a_variable = "hello, world!"
icxx"std::cout << $a_variable << std::endl;"

I am creating the code dynamically, e.g.

function test()
    a_variable = [4, 5]
    namespace = "std"
    cppcode = "$namespace::cout << \$(conversion_fn(a_variable)) << $namespace.endl;"
end

Here, cppcode is a string where some variables are interpolated ($namespace), while other variables (conversion_fn(a_variable)) are meant to be passed as arguments and thus should be interpolated by icxx_str, which also takes care of proper type conversion. conversion_fn takes care of converting Vector{Int64} to a type that icxx knows how to convert (namely: cxxt"std::vector<double>")

Now, I need to put variable cppcode as argument to icxx_str; however, if I do @icxx_str(cppcode) it obviously tries to use "cppcode" as string.

Thanks to the Slack chat, I came up with @eval @icxx_str($cppcode), which works if there are no local variables that must be interpolated by icxx_str, because @eval works in the global scope. Indeed, it works in the REPL, but not in a function!

How can I do?

2

Ok, right now I’m doing the following, which is very ugly and very pythonic (~= no metaprogramming):

function make_test_fn(namespace)
    eval(Meta.parse("""
        function test(a_variable=[4, 5])
            icxx"$namespace::cout << \$(conversion_fn(a_variable)) << $namespace.endl;"
        end
    """))
end

To ensure that the function is created at inclusion/import time:

for namespace in ["std", "other_name_space"]
    eval(Meta.parse("""
        function test(a_variable=[4, 5])
            icxx"$namespace::cout << \$(conversion_fn(a_variable)) << $namespace.endl;"
        end
    """))
end
3

Why not use Mustache.jl or any other template engine? It gives flexibility and there is no need to use eval(Meta.parse combination.

4

IMHO, Mustache is just too complex for simply defining a function. Julia should have the ability to run eval in the local scope, instead.

5

Julia does not have this ability because it would break everything that makes it fast, so it will probably never support it, it is a design decision.

You should not be using Meta.parse but instead the quote block. You are adding an unnecessary extra phase of syntax parsing.

Anyways, taking a step back and looking at the whole picture, I think that the right way to do that is using Julia to create C++ files, call the C++ compiler over them, and then use Cxx to link to the compiled libraries. See: Examples · Cxx.jl

3 Likes
6

There are many things that Julia can do and that lower the performance. Having the ability to run eval in local scope just adds an option for these somehow rare cases.

Sorry, I’m pretty new to Julia. Can you provide me an example?

That’s weird… Cxx is nice because allows to almost completely avoid worrying about compilation and C++ stuffs. After all, icxx is creating a C++ file and compiling, isn’t it?

1 Like
7

I would like to note a few things.

  1. The main issue is here is the confusion of compile-time and runtime. What you’re trying to do is a kind of double-compilation. First we create a Julian function, which generates C++ code, which then gets recompiled into a Julian expression and called. This workflow is a little unusual, and should really be carefully thought whether it is necessary.
  2. Having a local eval is very problematic for code optimization. This is not something that one can just turn on (easily) and say that the code should always run fast when no eval is ever touched. The rough idea is that having a local eval dramatically changes the guarantees one has in the code. Is it possible to introspect to make sure no local evals exist and do everything the current julian way to keep everything fast, while having local evals when necessary as a slow route. Probably. But it is probably not worth (and explicitly chosen against as a design feature) to do so.
  3. The recommended workflow I have for your particular problem is the following. Using string interpolation to generate the necessary C++ code entirely, then later in at the end of your script, generate the Meta.parse calls with Cxx.jl in global scope. In some sense, it is wrong to want to make it work in a function because it is the wrong “time” (see again the confusion of runtime and compile-time in 1)
1 Like
8

I think I did not express myself clearly enough. Having the possibility of executing eval in local scope would not make things slow only when you use this feature, it would make the whole language much slower, independently if this feature is used or not. Basically, the fact that, for example, Python allow things similar to eval in local scope that makes it much slower than Julia.

You should not be doing metaprogramming in Julia without reading the relevant section of the manual.

Probably. The problem here is that icxx is probably a macro because this forces the user to put the C++ code directly into the source, and then this step of C++ compilation and linking may be done in a precompiling phase. You are trying to bypass this by using eval and, therefore, it seems to me that your use case is not the use case that icxx focus on simplifying/solving. You would be better using low level machinery yourself if you want to generate C++ code on the fly because it seems icxx was not made to help in this specific situation.

1 Like
9

You’re right, the example here is confusing. In my real code, the function is generated at inclusion/import time.

I know about it. I have this problem though:

function_name = "my_function"
fn = quote
function $function_name()
   println("Hello")
end
end # quote
eval(fn)

creates error for the way function name is used

10

On the last problem. You can just

julia> function_name = "my_function";

julia> fnexpr = quote
           function $(Symbol(function_name))()
               println("Hello")
           end
       end;

julia> eval(fnexpr)
2 Likes
11

Perhaps we can suggest a better approach than the one you’re currently trying to use if you tell us what library you’re trying to use?

12

Well, the wrapper is almost completely done… by the way, it’s Essentia C++ library

13

Ok, and for the original example?

14

What is different in the original example for you to be in doubt?

15

The variable is interpolated inside a icxx"[stuffs]", that is: the interpolation must output a string with not quote, like in “$var”…

16

The quoting is to avoid the re-parsing of Julia code. The whole icxx literal is single Julia expression and can be assembled in separate and then interpolated inside the quote that avoids the re-parsing of the whole function structure.

17

icxx is a macro which wants a string, not an expression. the following doesn’t work (this is the problem described in the first post)

eval(quote
    icxx$cppcode
end)

eval(quote
    @icxx_str($cppcode)
end)
18

I meant building a string "@icxx_str(\"$(escape_string(cppcode))\")" and then interpolating it, i.e., interpolating the macro too. I have not a good environment to test it right now, does it not work? Anyway, this is just for performance, as you need to compile a C++ code behind the scenes (i.e., it is not just parsing a Julia code) the gain can be small.

19

Yes, I don’t think that there could be a lot of improvement since my function just calls an icxx macro. Also, having one string with all code inside it is more readable than splitting the body of the function into multiple statements. I think I will use the Meta.parse way for readability.

1 Like