Not able to load a module from custom system image

General Usage
, , ,
1

Dear experts,

I have recently started using Julia because of its nice ability to create a binary out of the code. Apologies for the long post, but I am trying to give you the full picture.

While I followed this documentation I managed to create a binary.
Now, my intention is to pass an array to this binary. As it is mentioned in the documentation, one can pass arguments by using the global variable ARGS.
I am not sure how this could help in getting/returning an array.

To be more specific, I would like to:

  1. write my algorithm in Julia (which gets an array, it does some calculations and returns a new array)
  2. do the precompile
  3. create the sysimage
  4. create the shared library
    and then call it in a similar way as in the documentation above.

While the above seemed to be quite tricky, I thought I could follow the example about embedding Julia in C and simply creating my own module. That didn’t work.

Here is what I tried:

I created the “my_test.jl”

module my_test

export real_main
export julia_main

function real_main(x::Float64, y::Float64)
        println("from main " , x, " " , y)
end

Base.@ccallable function julia_main(x::Float64, y::Float64)::Cint
    try
        real_main(x,y)
	return 0
    catch
        Base.invokelatest(Base.display_error, Base.catch_stack())
        return 1
    end
    return 0
end

if abspath(PROGRAM_FILE) == @__FILE__

    julia_main(3.,4.)
end

end

then I precompiled it, by using:

julia --startup-file=no --trace-compile=app_precompile.jl my_test.jl

Once the pre-compilation was successful, I created the create_sysimage.jl:

Base.init_depot_path()
Base.init_load_path()

@eval Module() begin
    Base.include(@__MODULE__, "my_test.jl")
    for (pkgid, mod) in Base.loaded_modules
        if !(pkgid.name in ("Main", "Core", "Base"))
            eval(@__MODULE__, :(const $(Symbol(mod)) = $mod))
        end
    end
    for statement in readlines("app_precompile.jl")
        try
            Base.include_string(@__MODULE__, statement)
        catch
            # See julia issue #28808
            Core.println("failed to compile statement: ", statement)
        end
    end
end # module

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

Then, I built the shared library based on that image, in 2 steps:

julia --startup-file=no -J"$JULIA_DIR/lib/julia/sys.so" --output-o sys.o create_sysimage.jl
gcc -g -shared -o libsys.so  -Wl,--whole-archive sys.o -Wl,--no-whole-archive -L"$JULIA_DIR/lib" -ljulia

Once this succeeds, I created the cpp file to use the library above. Therefore, my_test.cpp:

#include <julia.h>

JULIA_DEFINE_FAST_TLS()

int main()
{

        libsupport_init();
        jl_options.use_compiled_modules = JL_OPTIONS_USE_COMPILED_MODULES_YES;
        jl_options.image_file = JULIAC_PROGRAM_LIBNAME;
        jl_options.image_file_specified = 1;
        jl_init_with_image(NULL,JULIAC_PROGRAM_LIBNAME);
//Enabling the below gives a better explanation of te failure
/*
        jl_eval_string("using Main.my_test.jl");
        if (jl_exception_occurred()) {
                jl_call2(jl_get_function(jl_base_module, "showerror"),
                         jl_stderr_obj(),
                         jl_exception_occurred());
                jl_printf(jl_stderr_stream(), "\n");
                jl_atexit_hook(2);
                exit(2);
        }       
        jl_module_t* LA = (jl_module_t *)jl_eval_string("Main.my_test");
        if (jl_exception_occurred()) {
                jl_call2(jl_get_function(jl_base_module, "showerror"),
                         jl_stderr_obj(),
                         jl_exception_occurred());
                jl_printf(jl_stderr_stream(), "\n");
                jl_atexit_hook(3);
                exit(3);
        }
*/

        jl_function_t *func1 = jl_get_function(jl_main_module, "julia_main");
        if (jl_exception_occurred()) {
                jl_call2(jl_get_function(jl_base_module, "showerror"),
                         jl_stderr_obj(),
                         jl_exception_occurred());
                jl_printf(jl_stderr_stream(), "\n");
                jl_atexit_hook(4);
                exit(4);
        }
        jl_value_t* in1 = jl_box_float64(12.);
        jl_value_t* in2 = jl_box_float64(24.);
        jl_value_t* ret = NULL;

        JL_GC_PUSH3(&in1,&in2,&ret);

        ret = jl_call2(func1, in1, in2);

        JL_GC_POP();

        jl_atexit_hook(0);
}

And then compile it, as:

g++  -o pass_2_arrays_to_my_test_by_eval  -fPIC -I$JULIA_DIR/include/julia -L$JULIA_DIR/lib -ljulia  -L$CURRENT_FOLDER -lsys pass_2_arrays_to_my_test_by_eval.cpp  $JULIA_DIR/lib/julia/libstdc++.so.6

The JULIA_DIR points to the Julia’s installation directory and CURRENT_FOLDER points to the current working dir.

Calling the pass_2_arrays_to_my_test_by_eval fails with Segmentation Fault message.
To my understanding, it fails because it cannot load the module (you can see that if un-comment some lines in the cpp code).

Could someone give some help on that?
Some people in the past seem to do that without any issue (as here).

Thanks a lot in advance!

2 Likes
2

ok finally the solution is found!
The sysimage has to be:

Base.init_depot_path()
Base.init_load_path()

# the following 2 lines is to tell system image that MyApp shall be included in Main scope
push!(LOAD_PATH, pwd())
using my_test

@eval Module() begin
    for (pkgid, mod) in Base.loaded_modules
        if !(pkgid.name in ("Main", "Core", "Base"))
            eval(@__MODULE__, :(const $(Symbol(mod)) = $mod))
        end
    end
    for statement in readlines("app_precompile.jl")
        try
            Base.include_string(@__MODULE__, statement)
        catch
            # See julia issue #28808
            Core.println("failed to compile statement: ", statement)
        end
    end
end # module

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

and then it works.
Probably it is a dirty hack and in case there is something more, please paste it below.

3

HI, Vangelis

I am very new to juila and want to create a sysimage of my own module. I am able to create a sysimage for already available pacakges in julia. but If want to create a sysimage of local module ‘Foo’ its shows me an error that ‘packages Foo not in the project’ . I am in need of help to solve this problem

4

Hi Arvind,
Did you try the example for the sysimage?
Also, could you share your code and the steps you did?

5

I am using julia version 1.5.3 . I am just learning how to create a sysimage for our own module using Packagecompiler. I have read the Package compiler documentaion and checked with packages available in julia. But it was not working for own module. please help me out with this problem.
The code is used is below. I saved this in the name “ExtractApocalypse.jl”

module ExtractApocalypse
export ImPublicToo, callme
const IMPLEMENTATION_DETAIL = :no_worries_mate
callme() = IMPLEMENTATION_DETAIL
module ImPublicToo
export callmetoo
callmetoo() = :your_good
end
end

I am calling this module in another script and want to convert it into sysimage. The code is below. The name of this is “example_module.jl”

include("ExtractApocalypse.jl")
using .ExtractApocalypse
callme()# :no_worries_mate
ImPublicToo.callmetoo() # :your_good
IMPLEMENTATION_DETAIL # error
ExtractApocalypse.IMPLEMENTATION_DETAIL # error
using ExtractApocalypse: IMPLEMENTATION_DETAIL # error



using PackageCompiler
create_sysimage(:ExtractApocalypse, sysimage_path="sys_ExtractApocalypse.so", precompile_execution_file="example_module.jl")

In the above code i am able to call the module and execute it but while creating the sysimage i am getting an error that “ERROR: package(s) ExtractApocalypse not in project”. I am not sure how to tackle this problem. If anyone knows the solution please help me out.

6

Hi Arvind,
personally I tried a slightly different direction, outside julia terminal.
So, the steps below can be executed on your actual terminal.

First I do the pre-compile step, like this:
julia --startup-file=no --trace-compile=ApocalypseModule_precompile.jl ApocalypseModule.jl

Then, we need to prepare the custom_sysimage.jl file. In your case it should like that (I assume that you are using some other packages i.e. LinearAlgebra):

Base.reinit_stdio()
Base.init_depot_path()
Base.init_load_path()

push!(LOAD_PATH, pwd())
push!(LOAD_PATH, "/path/to/julia-1.5.3/share/julia/stdlib/v1.5/LinearAlgebra/src")
push!(LOAD_PATH, "/path/to/julia-1.5.3/share/julia/stdlib/v1.5/Libdl/src")
using PointsModule

@eval Module() begin
    using LinearAlgebra
    using ApocalypseModule
    for statement in readlines("ApocalypseModule_precompile.jll")
        try
            Base.include_string(@__MODULE__, statement)
        catch
            # See julia issue #28808
            Core.println("failed to compile statement: ", statement)
        end
    end
end # module

empty!(LOAD_PATH)
empty!(DEPOT_PATH)

Now that you have prepared the custom_sysimage.jl with the above lines, you can do:
julia --startup-file=no --output-o apocalypse_sys.o -J"/path/to/julia-1.5.3/lib/julia/sys.so" custom_sysimage.jl

The above command, generates a .o file, which later on can be passed to gcc and create a shared lib:
gcc -shared -o libapocalypse_sys.so -Wl,--whole-archive apocalypse_sys.o -Wl,--no-whole-archive -L"/path/to/julia/lib/" -ljulia

The most important step, is the precompilation. Make sure, that the ApocalypseModule_precompile.jl contains the statements that describe your Modules.

1 Like