Using struct created at runtime as argument type

General Usage
1

For a project I’m currently working on I have to do a lot of code generation. I run into a problem when I tried to create structs at runtime and use them in method definitions as argument types. It would be great if someone knows some kind of workaround for this problem or has an explanation for this behaviour. I created a minimal example to illustrate the problem.

main

push!(LOAD_PATH, @__DIR__)

using A

Core.eval(A, :(struct Foo end))

# B should be evaluated after A was modified
using B

A.jl

module A end

B.jl

module B
  using A
  function func1(value::A.Foo) end
end

Error message

ERROR: LoadError: UndefVarError: Foo not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
 [2] top-level scope at none:0
 [3] include at ./boot.jl:326 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1038
 [5] include(::Module, ::String) at ./sysimg.jl:29
 [6] top-level scope at none:2
 [7] eval at ./boot.jl:328 [inlined]
 [8] eval(::Expr) at ./client.jl:404
 [9] top-level scope at ./none:3
in expression starting at /home/xxx/Documents/julia-bug-01-min-example/B.jl:3
ERROR: LoadError: Failed to precompile B [top-level] to /home/xxx/.julia/compiled/v1.1/B.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1197
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853
 [6] include at ./boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at ./loading.jl:1038
 [8] include(::Module, ::String) at ./sysimg.jl:29
 [9] exec_options(::Base.JLOptions) at ./client.jl:267
 [10] _start() at ./client.jl:436
in expression starting at /home/xxx/Documents/julia-bug-01-min-example/main.jl:8
2

Hi Adriaan :wave:

I imagine there could be a more “Julian” way to do what you are trying to do. You might get more valuable help if you give a little more context around what you are trying to do.

That aside, it looks like your main (a hint you are coming from another language :blush:) is trying to create a struct Foo in a module A and your B is trying to dispatch on A.Foo, but A.Foo doesn’t exist when B is used.

I can think of two ways to proceed and there may be a better way still.

Option 1

One way to do this would be to create an AbstractFoo in A. Then, in your main you can just do

struct Foo <: A.AbstractFoo end

Then B can contain something like

func1(value::A.AbstractFoo) = error("Must implement `func1` for YourFoo <: AbstractFoo")

so you would implement

function func1(value::Foo)
    # do stuff with Foo
end

also in your main.

Your main would look like:

module Main

using A,B

struct Foo <: A.AbstractFoo end

function func1(value::Foo)
    # do stuff with Foo
end

Option 2

Inside A, you can create a method:

module A

function createfoo(args...)
    # Use `eval` to generate a Foo struct 
end

and inside B you can create a method

module B

function createfoofunc(args...)
    # use `eval` to generate a function to dispatch on your Foo
end

And then in your main you would do something like:

module Main

using A,B

A.createfoo(args...)

B.createfoofunc(args...)

end

Like I said, I suspect there is a better way. If you give some more context, it might help someone give a better answer :blush:

5 Likes
3

Hi Eric,

Thank you very much for your reply. The solutions you proposed definitely make sense to me. However I suspect that I can’t use a solution like this in my case. I guess I should have been more specific from the beginning, but to give more context now: I’m working on a plugin system for a simulation tool (written in C++), so that certain components can be developed with Julia. To provide the necessary C++ bindings I use the Julia C-API and I do something similar to the following code snippet within the initialization phase of the program.

jl_eval_string("using CppBindings");
jl_module_t* cppBindings = jl_eval_string("CppBindings");
add_methods_and_types(cppBindings);

Hereby CppBindings.jl is more or less an empty module and all its methods and types are generated by the C++ application. The purpose of these methods/types is apparently to wrap C++ methods/types.

When the simulation is running, specific actions can be hooked with Julia methods. This looks somehow like this:

module Plugin
  using CppBindings

  function julia_hook(arg)
    # No Problem to access generated types within method
    x = CppBindings.ArbitraryType(42)
    # No Problem to access generated methods within method
    y = CppBindings.arbitrary_method(1, 2, 3)
  end

  # For some reason using generated types in method definition doesn't work
  function another_julia_hook(arg::CppBindings.ArbitraryType) 
    ...
  end
end

Almost everything works as intended except that I can’t use types generated by the C++ application as argument types. Obviously being able to do this is crucial to extend functionality.

In the given scenario it’s unfortunately not possible to use your solutions because I don’t have a “main” Julia file as an entry point. Instead everything is put together by the C++ application. For reasonable modularization it’s also important that I can use these types as argument types within seperate module files.

As you already recognized I’m relatively new to Julia :sweat_smile: so I have a hard time judging whether this behaviour is to be expected or if this is some kind of bug.

4

Ah, but you do have a main. It is called Plugin in this case :blush:

I have no idea about C++ bindings, but I still think something along the lines of Option 2 could work for you.

I “think” you can write Julia methods with inputs that are driven by C++ but do the metaprogramming within Julia. It almost seems like you are trying to do Julia metaprogramming entirely from C++ :sweat_smile: I think there are only a handful or mortals that could get that to work and I am not one of them :sweat_smile:

Sounds like this is over my head. Hope someone else chimes in :pray:

5

I “think” you can write Julia methods with inputs that are driven by C++ but do the metaprogramming within Julia. It almost seems like you are trying to do Julia metaprogramming entirely from C++ :sweat_smile:

Yes it’s probably best practice to to encapsulate the functionality into Julia methods and call these from C++. For the most part I’m already doing that :wink:

Anyway I came up with a solution to the problem. I had to deactivate precompilation in modules that depend on types generated at runtime. Otherwise an unmodified version of the module is used in the precompilation phase. The example I posted above works fine if change it this way:

__precompile__(false)
module B
  using A
  function func1(value::A.Foo) end
end

In general eval shouldn’t be used on already defined modules to prevent such problems in the first place (in some cases the compiler will also show a warning). However there might be exceptions where it’s not possible to avoid it and the only solution is to deactivate precompilation for specific modules.

1 Like