Difficulties putting code in module (on an M1 machine) (pre-compilation, cfunctions, global variables)

General Usage
, ,
1

I’m building an interactive simulation of a Spin Ising model and I’ve been experiencing some difficulties. I’m using QML to build an interface. Before, I had everything in a large script with a lot of global variables, and I’ve been trying to get everything neatly into a module, with all parameters in a struct Sim, which can be initialised when the module is used. The struct also can be called as a function to start the simulation interface. One problem I’m facing is that the QML packages (at least the way I’m using it) needs a c function as a method to update an image that is displayed. This c function is initialised as follows:

function showlatest(buffer::Array{UInt32, 1}, width32::Int32, height32::Int32)
    buffer = reshape(buffer, size(img[]))
    buffer = reinterpret(RGB, buffer)
    buffer .= img[]
    return
end

showlatest_cfunction = CxxWrap.@safe_cfunction(showlatest, Cvoid, 
                                               (Array{UInt32,1}, Int32, Int32))

The CxxWrap.@safe_cfunction is defined as

macro safe_cfunction(f, rt, args)
  return esc(:($(@__MODULE__).SafeCFunction(@cfunction($f, $rt, $args), $rt, [$(args.args...)])))
end

where

struct SafeCFunction
  fptr::Union{Ptr{Cvoid},Base.CFunction}
  return_type::Type
  argtypes::Array{Type,1}
end

In the showlatest function, there is a variable named img, which should be a Ref to a matrix. Ideally, I would store this variable only in my struct Sim, and define the SafeCFunction after a user has initialised an instance of the Sim struct, which should then have a local copy of the img reference. However, the macro CxxWrap.@safe_cfunction, is of course run at compile time, so doing something like

function create_showlatest_cfunc(sim::Sim)
          img = sim.img
  function showlatest
    ...
  end
  
  showlatest_cfunction = CxxWrap.@safe_cfunction(showlatest, Cvoid, 
                                                 (Array{UInt32,1}, Int32, Int32))

end

gives the error that showlatest is not defined. I tried interpolating showlatest into the macro, e.g.:

...
showlatest_cfunction = CxxWrap.@safe_cfunction($showlatest, Cvoid, 
                                                 (Array{UInt32,1}, Int32, Int32))

but as it turns out, since I’m on an M1 machine, closures aren’t supported on the apple arm architecture, so this won’t work.

Before I had everything in a module, I would just have a global const variable called img and everything worked out.

What I tried now is just having a const img in the global scope of my module, and storing the same ref inside my struct. This way Sim should only be instantiated once, but that’s fine for my purposes. At first this seemed to be working, but at the time, my module wasn’t compiling. Now it’s compiling, but as soon as I run my code (I think the moment the cfunction is called actually, but it’s hard to confirm this), I get an error and my REPL in vscode crashes and disappears before I can read it. Screen recording it reveals this error:

signal (10): Bus error: 10
in expression starting at none:0
unknown function (ip: 0x1757b46b0
...

This happens when I make my module files visible using `push!(LOAD_PATH, pwd())` and then `using IsingSim` in my main file. When I instead import the file into my main file, and `using .IsingSim`, I get a different error:

libc++abi: terminating with uncaught exception of type std::runtime_error: Could not find Module QML when looking up function get_julia_call

signal (6): Abort trap: 6
in expression starting at none:0
__pthread_kill at /usr/lib/system/libststem_kernel.dylib (unknown line)
…

I tried using @eval to define img in local scope

function showlatesteval(sim)
    img = sim.img
    function showlatest(buffer::Array{UInt32, 1}, width32::Int32, height32::Int32)
        buffer = reshape(buffer, size(img))
        buffer = reinterpret(ARGB32, buffer)
        buffer .= img
        return
    end

    @eval $:(CxxWrap.@safe_cfunction($showlatest, Cvoid, (Array{UInt32,1}, Int32, Int32)))
end


startSim(sim)
  showlatest_cfunction = showlatesteval(sim)
  ...
end

This gets me a bit further, and doesn’t throw the same error. I even see an image displayed, at first. However, my image should be (500,500), and I soon get one of two errors

UndefRefError: access to undefined reference

DimensionMismatch: new dimensions (10761626400, 4390640200) must be consistent with array size 250000

So it seems that my reference is getting invalidated somehow, though I don’t understand how.

I’m kind of at a loss right now, and don’t know how to continue. Any help would greatly be appreciated.

My full code can be found here

2

Without getting into too much detail, it sounds like you need to move your initialization-time code into the __init__() function, as described here: Modules · The Julia Language

1 Like
3

Thanks a lot! Putting the showlatest_cfunction in the init function, keeping everything else in the global scope of my module, works!

I’m still not sure why the @eval trick didn’t work though.

4

Actually, coming back to this, there still appears to be a weird problem. I found this out because I got a new machine on which I did a clean install. I was getting the following problem again:

libc++abi: terminating with uncaught exception of type std::runtime_error: Could not find module QML when looking up function get_julia_call

signal(6): Abort trap: 6
In expression starting at none:0
__pthread_kill at /usr/lib/system/libststem_kernel.dylib (unknown line)
…

I was using the exact same code as on my old machine, on which the code executed without problem. The only difference was that I didn’t make a sysimage yet. Making a sysimage on the new machine fixes the problem. I found out that adding the line using QML in the main file where I include my own modules also fixed the problem (I already use QML in the module as well, of course). Naturally I want my module to be self contained, so this isn’t a great solution. What am I doing wrong?

5

I think that the problem is, that the external QML library (which is a c++ library) expects some module QML to be defined in the Main module, with the function get_julia_call to be there. However, since I’ve wrapped everything in my own module, the QML module is of course not defined in the Main module anymore.

I tried adding the line export QML in my own module and this seems to be doing the trick. It seems hacky though, so is there a better solution to do this?