Hi all,
I’ll briefly explain my specific problem, then describe the general question with a minimal working example.
The specific problem
I’m interested in using Julia to create an LV2 plugin, which is basically an effect for audio processing (e.g. reverb, distortion, echo). The general setup is that there’s a “host” program which runs and coordinates multiple “plugins”, which each represent an individual transformation of the audio stream.
Both hosts and plugins are generally written in C. The plugin is compiled into a shared object library (.so), and the host calls specific functions from the plugin via dlopen and dlsym in accordance with the LV2 API. There’s a standard library for writing hosts called Livl.
So I want to write a C plugin with embedded Julia doing the heavy lifting. The issue is that hosts call dlopen via Livl here with only the RTLD_NOW flag, not the RTLD_GLOBAL flag, whereas the Julia doc page on embedding says very clearly
Currently, dynamically linking with the
libjuliashared library requires passing theRTLD_GLOBALoption.
Is it possible to write a plugin with embedded julia without modifying the host code?
Minimal working example
I’ve created the following MWE, which is also available on github, to represent the situation. In this example, we have a host calling one function from a plugin library which contains embedded Julia. It suceeds when the host uses RTLD_NOW, but fails otherwise.
plugin.c
#include <stdio.h>
#include <julia.h>
float plugin_func(float x)
{
jl_function_t *julia_sqrt;
jl_value_t *julia_input, *julia_output;
float result;
jl_init();
printf("Plugin - Getting julia function\n");
julia_sqrt = jl_get_function(jl_main_module, "sqrt");
printf("Plugin - Boxing input\n");
julia_input = jl_box_float32(x);
printf("Plugin - Calling julia function.\n");
julia_output = jl_call1(julia_sqrt, julia_input);
printf("Plugin - Unboxing output\n");
result = jl_unbox_float32(julia_output);
printf("Plugin - Returning result\n");
return result;
}
host.c
#include <stdio.h>
#include <dlfcn.h>
int main() {
void *plugin_lib;
float(*plugin_func)(float x);
float plugin_result;
printf("Host - dlopen\n");
plugin_lib = dlopen("./plugin.so", RTLD_NOW);
printf("Host - dlsym\n");
plugin_func = dlsym(plugin_lib, "plugin_func");
printf("Host - call plugin_func\n");
plugin_result = plugin_func(9.0f);
printf("Host - Plugin returned %.2f\n", plugin_result);
printf("Host - dlclose\n");
dlclose(plugin_lib);
printf("Host - finished\n");
return 0;
}
Makefile
JFLAGS=$(shell $(shell dirname $(shell which julia))/../share/julia/julia-config.jl --cflags --ldflags --ldlibs)
CC=gcc
host: host.c plugin.so
$(CC) -std=c99 -Wall -ldl $< -o $@
%.so: %.c
$(CC) -shared -o $@ $(JFLAGS) -fPIC $<
.PHONY: clean
clean:
rm -f *.so host
This example works fine when the host uses RTLD_GLOBAL in dlopen:
$ make
gcc -shared -o plugin.so -std=gnu99 -I'/home/oliver/local/src/julia-1.4.0/include/julia' -fPIC -L'/home/oliver/local/src/julia-1.4.0/lib' -Wl,--export-dynamic -Wl,-rpath,'/home/oliver/local/src/julia-1.4.0/lib' -Wl,-rpath,'/home/oliver/local/src/julia-1.4.0/lib/julia' -ljulia -fPIC plugin.c
gcc -std=c99 -Wall -ldl host.c -o host
$ ./host
Host - dlopen
Host - dlsym
Host - call plugin_func
Plugin - Getting julia function
Plugin - Boxing input
Plugin - Calling julia function.
Plugin - Unboxing output
Plugin - Returning result
Host - Plugin returned 3.00
Host - dlclose
Host - finished
But without RTLD_GLOBAL, it fails because it can’t access the necessary symbols from libjulia.
$ sed -i 's/RTLD_NOW | RTLD_GLOBAL/RTLD_NOW/' host.c
$ make
gcc -std=c99 -Wall -ldl host.c -o host
$ ./host
Host - dlopen
Host - dlsym
Host - call plugin_func
fatal: error thrown and no exception handler available.
ErrorException("could not load symbol "jl_n_threads":
./host: undefined symbol: jl_n_threads")
jl_errorf at /buildworker/worker/package_linux64/build/src/rtutils.c:77
jl_dlsym at /buildworker/worker/package_linux64/build/src/dlload.c:269
jl_load_and_lookup at /buildworker/worker/package_linux64/build/src/runtime_ccall.cpp:63
nthreads at ./threadingconstructs.jl:19 [inlined]
__preinit_threads__ at ./task.jl:518
jfptr___preinit_threads___1959.clone_1 at /home/oliver/local/src/julia-1.4.0/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2144 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2322
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1692 [inlined]
_julia_init at /buildworker/worker/package_linux64/build/src/init.c:765
jl_init_with_image__threading at /buildworker/worker/package_linux64/build/src/jlapi.c:74
jl_init__threading at /buildworker/worker/package_linux64/build/src/jlapi.c:102
plugin_func at ./plugin.so (unknown line)
main at ./host (unknown line)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at ./host (unknown line)
atexit hook threw an error: ErrorException("could not load symbol "jl_array_del_beg":
./host: undefined symbol: jl_array_del_beg")
The general question
I’m not very familiar with compiler/linker details, so forgive my ignorance. But my question becomes:
Is there any way around the RTLD_GLOBAL requirement? Is it somehow possible to statically compile libjulia into the shared library? Or if not, is it possible to compile a static library/executable which can be dlopen’d and dlsym’d in the same way a shared library would?
If this is just a current Julia limitation and there’s absolutely no way around it, I’d at least be interested to learn more about the issue and what causes the current dependence on RTLD_GLOBAL.
Thanks!
Oliver