Wrapping a Complicated, Multi-Threaded C/C++ Library

Internals & Design
,
1

I’m attempting to wrap a somewhat complicated C++ library using CxxWrap and am running into issues when it comes to blocking and multi-threaded C++ code. Here’s an abbreviated example of the problem:

# Functions/objects prefixed with cpp_/Cpp are 
# trivial wrappers around their C++ equivalents.
function loop(object)
    while cpp_is_ready(object)
        cpp_pthread_cond_wait(object) # Calls pthread_cond_wait in C++
        msg = cpp_get_message(object)
        # Do stuff with msg
    end
end

function my_callback(msg)
    # Do stuff with msg
end

function main()
    object = CppObject(@cfunction(my_callback, Nothing, (Cstring, )))
    start(object) # Starts a pthread in C++ 
    Threads.@spawn reader(object)
    request(object, "foobar")
end

At a high level, this is a simple reader/writer design. start(object) calls a C++ function that initializes a writer thread (using pthread) that writes a message to an internal queue for each call to request(object, msg_request). reader(object) waits for a message via cpp_pthread_cond_wait(object) which internally calls pthread_cond_wait in C++ land. process_message then grabs that message from the queue and calls the callback my_function.

All together, we have three threads:
Thread 1: pthread created in C++ from call to start(object)
Thread 2: Julia thread created from @spawn
Thread 3: Main Julia thread that makes the call to request(object).

and one lock, the pthread_mutex created in C++ land.

Now this all works fine, until an exception occurs, either from an error or something like Ctrl-C. When that occurs, the process hangs and the terminal becomes unresponsive to input. Not even Ctrl-C (SIGINT) or Ctrl-\ (SIGQUIT) works. I have to close the terminal. Note that issue only occurs when mixing Julia and C++, if I recreate the above code block in pure C++ then Ctrl-C works as expected.

Reading through the Julia manual, I found this discussion next to the docs for @threadcall:

If a C library performs a blocking operation, that prevents the Julia scheduler from executing any other tasks until the call returns.

Which leads me to suspect that the issue is unrelated to the multi-threading and solely to do with the call to cpp_pthread_cond_wait(object). Given that the pure C++ version of the above code works find, I suspect that SIGINT doesn’t actually get issued so the reader that’s waiting on cpp_pthread_cond_wait never wakes up. I though about using @threadcall, but it is has this caveat:

It is very important that the called function does not call back into Julia, as it will segfault.

Which disqualifies my since the library requires that I pass it a callback. With this in mind, I tried the following modification:

function main(object)
    object = CppObject(@cfunction(my_callback, Nothing, (Cstring, )))
    try
        start(object) # Starts a pthread in C++ 
        Threads.@spawn reader(object)
        request(object, "foobar")
    catch e
        # Issue a signal that wakes up the waiting thread in `cpp_pthread_cond_wait`
        issue_signal(object)
    end
end

which surprisingly fixes the issue! But it seems like such an ugly hack. So here’s my question:

  1. Is my deduction that the root cause of the issue is the call to cpp_pthread_cond_wait correct/probable?
  2. What’s a better solution to the above problem? Can I make sure that SIGINT gets signaled somehow? Would that even solve the issue? What about calling jl_yield in cpp_pthread_cond_wait from C++?

Thank you for any and all help on an incredibly frustrating problem!

1 Like