Is it possible for "ccall" to call the shared library created by PackageCompiler.jl?

lionisxn · July 16, 2022, 3:14am

I am trying to create a .dll that can be called by other languages, e.g. MATLAB or C. I just test the example given in “PackageCompiler.jl”, and generate the following file bundles where “fist_lib_wy.dll” is the created dll.

When I tried to call for “fist_lib_wy.dll” through the following codes in a script:

const libnm = “D:/MyLib/MyLibCompiled/bin/fist_lib_wy”

ccall((:increment64, libnm), Clong, (Clong, ), 1)

The following crash errors emerge:
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x0 – unknown function (ip: 0000000000000000)
in expression starting at D:\瀛︿範璧勬枡\endnote鏂
囦欢搴揬绗旇\璁烘枃鍖匼1 鍒濈鏈姇\ACDCpower\DEBUGDATA.jl:25
unknown function (ip: 0000000000000000)
increment64 at D:\MyLib\MyLibCompiled\bin\fist_lib_wy.DLL (unknown line)
top-level scope at D:\瀛︿範璧勬枡\endnote鏂囦欢搴揬绗旇\璁烘枃鍖匼1 鍒濈鏈姇\ACDCpower\DEBUGDATA.jl:25
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:876
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:830
jl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:894 [inlined]
jl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:944
eval at .\boot.jl:373 [inlined]
include_string at .\loading.jl:1196
include_string at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\utils.jl:286 [inlined]
#218 at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\eval.jl:121
withpath at D:\Program Files\Julia.julia\packages\CodeTools\VsjEq\src\utils.jl:30
unknown function (ip: 0000000000f0ac04)
withpath at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\eval.jl:9
#217 at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\eval.jl:119
unknown function (ip: 0000000000f0a3f3)
with_logstate at .\logging.jl:511
with_logger at .\logging.jl:623 [inlined]
#216 at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\eval.jl:118 [inlined]
hideprompt at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\repl.jl:127
macro expansion at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\eval.jl:117 [inlined]
macro expansion at D:\Program Files\Julia.julia\packages\Media\ItEPc\src\dynamic.jl:24 [inlined]
eval at D:\Program Files\Julia.julia\packages\Atom\BEJkS\src\eval.jl:114
unknown function (ip: 0000000000f043c3)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:713#invokelatest#2 at .\essentials.jl:716
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [Starting Julia…

I have no idea what I missed in the compiling or calling process. I am eager for an answer from the Julia community. Many thanks

mkitti · July 16, 2022, 6:39am

Do you mean this example?
https://julialang.github.io/PackageCompiler.jl/stable/libs.html

It looks like you modified it slightly. Can you show us the modifications?

mkitti · July 16, 2022, 7:01am

I just realized that you are trying to call this from within Julia. I do not think this will work. For that, try loading the library as a system image.

Otherwise, what you did do is meant to be used by a C program outside of Julia.

lionisxn · July 16, 2022, 7:45am

hi, nkitti! Thank your for your guid. The modifications are made as follows:

The folder of “mylib” is firstly copied to D:\
run the code in a script “create_library(“D:/MyLib”, “MyLibCompiled”; lib_name=“fist_lib_wy”, header_files = [“D:/MyLib/build/mylib.h”], force = true, precompile_execution_file = [“D:/MyLib/build/generate_precompile.jl”], precompile_statements_file=[“D:/MyLib/build/additional_precompile.jl”], incremental = false, filter_stdlibs = true)”
execute " const libnm = “D:/MyLib/MyLibCompiled/bin/fist_lib_wy”

ccall((:increment64, libnm), Clong, (Clong, ), 1) "

After the step 3), the above errors are thrown and the REPL crashed

lionisxn · July 16, 2022, 7:48am

Do you mean that the shared library created by Julia can not be called by “ccall”, and only the system image can do so? I have tried to call it through MATLAB by executing “calllib”, and it crashed also.

mkitti · July 16, 2022, 8:08am

Did you initialize Julia before trying to do so?

mkitti · July 16, 2022, 8:12am

Take a look at libcg/main.c at master · simonbyrne/libcg · GitHub

lionisxn · July 16, 2022, 8:26am

I tried to initial julia using the syntax: calllib(‘fist_lib_wy’, ‘init_julia’, int32(4), {‘’}); However，matlab (ver. 2021b) crashed also. Could you please post a tutorial on how to call the examplified shared library in the binary of PackageCompiler through MATLAB? It will be great helpful to fully understand the integration of Julia with other platform.

thautwarm · July 16, 2022, 9:17am

Currently, calling the shared libraries generated by PackageCompiler.jl from Julia does not work. I’m also looking forward to this.

lionisxn · July 16, 2022, 10:54am

I just thought that a shared library created by Julia might be a standard one that can be called for by any language which supports c shared library. Am I wrong?

ohmsweetohm1 · July 20, 2022, 2:52pm

Any solution to this? I’m having the same problem.

ohmsweetohm1 · July 20, 2022, 3:41pm

For instance if I want to call this from Matlab I have to load the library via loadlibrary. I can load the mylib and find increment32 and increment64 in it. But I get an error when directly calling them. As I understand I have to init Julia before calling, but where do I find the init_julia and shudown_julia?

>> addpath('<path>\MyLibCompiled\bin')
>> loadlibrary('mylib', '<path>\MyLibCompiled\include\mylib.h')
>> libfunctions('mylib')

Functions in library mylib:

increment32  increment64

This crashes Matlab

>> calllib('mylib', 'increment64', 1)


Stack Trace (from fault):
[  0] 0x0000000004cb370f <path>\MyLibCompiled\bin\libjulia-internal.dll+00931599 jl_exception_handler+00000031
[  1] 0x0000000004d672e2 <path>\MyLibCompiled\bin\libjulia-internal.dll+01667810 _julia_personality+00000034
[  2] 0x00007ffedfef229f                      C:\WINDOWS\SYSTEM32\ntdll.dll+00664223 _chkstk+00000287
[  3] 0x00007ffedfea1454                      C:\WINDOWS\SYSTEM32\ntdll.dll+00332884 RtlRaiseException+00001076
[  4] 0x00007ffedfef0dce                      C:\WINDOWS\SYSTEM32\ntdll.dll+00658894 KiUserExceptionDispatcher+00000046

ohmsweetohm1 · July 20, 2022, 4:11pm

It is not in mylib:

>> loadlibrary('mylib', '<path>\MyLibCompiled\include\mylib.h', 'addheader', '<path>\MyLibCompiled\include\init_julia.h')
>> calllib('mylib', 'init_julia', 0, [''])
Error using calllib
Method was not found.

mkitti · July 20, 2022, 6:24pm

You still need the Julia runtime, meaning you need the various Julia shared libraries in addition to the one you created. For example, julia.dll or libjulia.so.1.7.

ohmsweetohm1 · July 20, 2022, 7:00pm

What do you mean I need those? PackageCompiler.jl bundled them in the bin folder

BTW: I’m on Windows

ohmsweetohm1 · July 20, 2022, 7:03pm

init_julia is also not from Julia itself but from ‘PackageCompiler.jl’

github.com

JuliaLang/PackageCompiler.jl/blob/d902a236a3d90bdcb5867967d73df4fd259584f1/src/julia_init.c#L64


      
          #ifdef _WIN32
              _putenv_s("JULIA_DEPOT_PATH", share_dir);
              _putenv_s("JULIA_LOAD_PATH", share_dir);
          #else
              setenv("JULIA_DEPOT_PATH", share_dir, 1);
              setenv("JULIA_LOAD_PATH", share_dir, 1);
          #endif
              free(share_dir);
          }
          
          
void init_julia(int argc, char **argv) {
              setup_args(argc, argv);
          
          
    const char *sysimage_path = get_sysimage_path(JULIAC_PROGRAM_LIBNAME);
              char *_sysimage_path = strdup(sysimage_path);
              char *root_dir = dirname(dirname(_sysimage_path));
              set_depot_load_path(root_dir);
              free(_sysimage_path);
          
          
    jl_options.image_file = sysimage_path;
              julia_init(JL_IMAGE_CWD);

ohmsweetohm1 · July 20, 2022, 8:22pm

I guess PackageCompiler.jl compiles the two methods together with the @ccallable functions into mylib.

github.com

JuliaLang/PackageCompiler.jl/blob/d902a236a3d90bdcb5867967d73df4fd259584f1/src/PackageCompiler.jl#L522


      
                                  base_sysimage,
                                  precompile_execution_file,
                                  precompile_statements_file,
                                  cpu_target,
                                  script,
                                  sysimage_build_args,
                                  extra_precompiles,
                                  incremental)
          object_files = [object_file]
          if julia_init_c_file !== nothing
              push!(object_files, compile_c_init_julia(julia_init_c_file, basename(sysimage_path)))
          end
          create_sysimg_from_object_file(object_files,
                                      sysimage_path;
                                      compat_level,
                                      version,
                                      soname)
          
          
rm(object_file; force=true)
          
          
if Sys.isapple()

But why does Matlab complain than?

lionisxn · July 20, 2022, 11:59pm

I just successfully compiled my project by following the example (GitHub - simonbyrne/libcg). A lot of time was wasted since I was not familiar with the makefile, or essentially, the ‘gcc’ command. After this, I recognized that a julia shared library can not be called without the support of Julia engine, that is why the init_julia() must be called prior to any shared function. So, I have an idea to make julia shared libary callable by matlab or another language other than C. That is we can compile a julia shared library to a C standard shared library via “gcc”, and edit a head file using C or C++. I am ready to test it, wish good luck!

lionisxn · July 21, 2022, 12:08am

It seems that MATLAB can not directly call a Julia shared library. But I am not sure about this. You can explore it further. I can share a reminder that never forget to add the share/julia/bin to the PATH environment variable if you were using Windows. I’m looking forward to your good news.

mkitti · July 21, 2022, 12:31am

Yes, but as you can see init_julia calls julia_init from julia.h:

github.com

JuliaLang/julia/blob/6008cbc019424a53786552a511bbef92099c033a/src/julia.h#L1730


      
              }
          
          // initialization functions
          typedef enum {
              JL_IMAGE_CWD = 0,
              JL_IMAGE_JULIA_HOME = 1,
              //JL_IMAGE_LIBJULIA = 2,
          } JL_IMAGE_SEARCH;
          
          JL_DLLEXPORT const char *jl_get_libdir(void);
          JL_DLLEXPORT void julia_init(JL_IMAGE_SEARCH rel);
          JL_DLLEXPORT void jl_init(void);
          JL_DLLEXPORT void jl_init_with_image(const char *julia_bindir,
                                               const char *image_path);
          JL_DLLEXPORT const char *jl_get_default_sysimg_path(void);
          JL_DLLEXPORT int jl_is_initialized(void);
          JL_DLLEXPORT void jl_atexit_hook(int status);
          JL_DLLEXPORT void jl_postoutput_hook(void);
          JL_DLLEXPORT void JL_NORETURN jl_exit(int status);
          JL_DLLEXPORT const char *jl_pathname_for_handle(void *handle);

Binary dependency searches are a bit strange on Windows. This is why the DLLs are in the bin directory rather than a lib directory like on the other operating systems. However, the search process is also well documented:

Here is a key excerpt:

If SafeDllSearchMode is enabled, the search order is as follows:

The directory from which the application loaded.

The system directory. Use the GetSystemDirectory function to get the path of this directory.

The 16-bit system directory. There is no function that obtains the path of this directory, but it is searched.

The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.

The current directory.

The directories that are listed in the PATH environment variable. Note that this does not include the per-application path specified by the App Paths registry key. The App Paths key is not used when computing the DLL search path.

On Windows, Julia itself uses option #1 above. The library and the executables are located in the same bin directory. When you launch MATLAB, it is not clear if Julia’s bin directory is either the “The current directory” for option #5 or on the PATH for option #6.

Note that the path relative to the library being loaded is not one the standard search path.

One approach on Windows is that you can explicitly load the dependencies first before loading the library of interest. This takes advantage of the following feature:

If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL.

Given the number of shared libraries that need to be loaded, this might become very tedious. Thus, I recommend check the PATH environment variable within MATLAB to make sure the relevant directory is on it. Alternatively, change MATLAB’s current working directory to the relative path with cd.