Not able to load a module from custom system image

Dear experts,

I have recently started using Julia because of its nice ability to create a binary out of the code. Apologies for the long post, but I am trying to give you the full picture.

While I followed this documentation I managed to create a binary.
Now, my intention is to pass an array to this binary. As it is mentioned in the documentation, one can pass arguments by using the global variable ARGS.
I am not sure how this could help in getting/returning an array.

To be more specific, I would like to:

  1. write my algorithm in Julia (which gets an array, it does some calculations and returns a new array)
  2. do the precompile
  3. create the sysimage
  4. create the shared library
    and then call it in a similar way as in the documentation above.

While the above seemed to be quite tricky, I thought I could follow the example about embedding Julia in C and simply creating my own module. That didn’t work.

Here is what I tried:

I created the “my_test.jl”

module my_test

export real_main
export julia_main

function real_main(x::Float64, y::Float64)
        println("from main " , x, " " , y)
end

Base.@ccallable function julia_main(x::Float64, y::Float64)::Cint
    try
        real_main(x,y)
	return 0
    catch
        Base.invokelatest(Base.display_error, Base.catch_stack())
        return 1
    end
    return 0
end

if abspath(PROGRAM_FILE) == @__FILE__

    julia_main(3.,4.)
end

end

then I precompiled it, by using:

julia --startup-file=no --trace-compile=app_precompile.jl my_test.jl

Once the pre-compilation was successful, I created the create_sysimage.jl:

Base.init_depot_path()
Base.init_load_path()

@eval Module() begin
    Base.include(@__MODULE__, "my_test.jl")
    for (pkgid, mod) in Base.loaded_modules
        if !(pkgid.name in ("Main", "Core", "Base"))
            eval(@__MODULE__, :(const $(Symbol(mod)) = $mod))
        end
    end
    for statement in readlines("app_precompile.jl")
        try
            Base.include_string(@__MODULE__, statement)
        catch
            # See julia issue #28808
            Core.println("failed to compile statement: ", statement)
        end
    end
end # module

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

Then, I built the shared library based on that image, in 2 steps:

julia --startup-file=no -J"$JULIA_DIR/lib/julia/sys.so" --output-o sys.o create_sysimage.jl
gcc -g -shared -o libsys.so  -Wl,--whole-archive sys.o -Wl,--no-whole-archive -L"$JULIA_DIR/lib" -ljulia

Once this succeeds, I created the cpp file to use the library above. Therefore, my_test.cpp:

#include <julia.h>

JULIA_DEFINE_FAST_TLS()

int main()
{

        libsupport_init();
        jl_options.use_compiled_modules = JL_OPTIONS_USE_COMPILED_MODULES_YES;
        jl_options.image_file = JULIAC_PROGRAM_LIBNAME;
        jl_options.image_file_specified = 1;
        jl_init_with_image(NULL,JULIAC_PROGRAM_LIBNAME);
//Enabling the below gives a better explanation of te failure
/*
        jl_eval_string("using Main.my_test.jl");
        if (jl_exception_occurred()) {
                jl_call2(jl_get_function(jl_base_module, "showerror"),
                         jl_stderr_obj(),
                         jl_exception_occurred());
                jl_printf(jl_stderr_stream(), "\n");
                jl_atexit_hook(2);
                exit(2);
        }       
        jl_module_t* LA = (jl_module_t *)jl_eval_string("Main.my_test");
        if (jl_exception_occurred()) {
                jl_call2(jl_get_function(jl_base_module, "showerror"),
                         jl_stderr_obj(),
                         jl_exception_occurred());
                jl_printf(jl_stderr_stream(), "\n");
                jl_atexit_hook(3);
                exit(3);
        }
*/

        jl_function_t *func1 = jl_get_function(jl_main_module, "julia_main");
        if (jl_exception_occurred()) {
                jl_call2(jl_get_function(jl_base_module, "showerror"),
                         jl_stderr_obj(),
                         jl_exception_occurred());
                jl_printf(jl_stderr_stream(), "\n");
                jl_atexit_hook(4);
                exit(4);
        }
        jl_value_t* in1 = jl_box_float64(12.);
        jl_value_t* in2 = jl_box_float64(24.);
        jl_value_t* ret = NULL;

        JL_GC_PUSH3(&in1,&in2,&ret);

        ret = jl_call2(func1, in1, in2);

        JL_GC_POP();

        jl_atexit_hook(0);
}

And then compile it, as:

g++  -o pass_2_arrays_to_my_test_by_eval  -fPIC -I$JULIA_DIR/include/julia -L$JULIA_DIR/lib -ljulia  -L$CURRENT_FOLDER -lsys pass_2_arrays_to_my_test_by_eval.cpp  $JULIA_DIR/lib/julia/libstdc++.so.6

The JULIA_DIR points to the Julia’s installation directory and CURRENT_FOLDER points to the current working dir.

Calling the pass_2_arrays_to_my_test_by_eval fails with Segmentation Fault message.
To my understanding, it fails because it cannot load the module (you can see that if un-comment some lines in the cpp code).

Could someone give some help on that?
Some people in the past seem to do that without any issue (as here).

Thanks a lot in advance!

1 Like

ok finally the solution is found!
The sysimage has to be:

Base.init_depot_path()
Base.init_load_path()

# the following 2 lines is to tell system image that MyApp shall be included in Main scope
push!(LOAD_PATH, pwd())
using my_test

@eval Module() begin
    for (pkgid, mod) in Base.loaded_modules
        if !(pkgid.name in ("Main", "Core", "Base"))
            eval(@__MODULE__, :(const $(Symbol(mod)) = $mod))
        end
    end
    for statement in readlines("app_precompile.jl")
        try
            Base.include_string(@__MODULE__, statement)
        catch
            # See julia issue #28808
            Core.println("failed to compile statement: ", statement)
        end
    end
end # module

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

and then it works.
Probably it is a dirty hack and in case there is something more, please paste it below.