Struggling with ccall

Dear Julia developers,
It’s almost a week I’m trying to call a custom C (or C++) function in Julia.
In particular what I want is to call a C function I wrote which takes some matrices as arguments and modify one of these. I still didn’t understand if I need to return it or since I give it a pointer it should modify it. Btw, I’m not able to do this task and I’ve tried everything. Your help is REALLY appreciated.

In detail, the function I wrote in C is:
ccode.c

#include <math.h>
#include <string.h>
 //#include <stdio.h>
void prob( double **p, double **a, double **b, int na, int nb) {
    for (int i=0; i<na; i++){
        double pr=a[i][0];
        for (int n=0; n<nb; n++){
            double pr_n=pr;
            for (size_t l=0; l < sizeof(b[n]); l++){
                pr_n=pr_n+b[n][l]*a[i][l+1];
            };
            pr_n=1/(1+exp(-pr_n));
            p[i][n]=pr_n;
        };
    };
};

then I run with no errors:
>g++ -o ccode.so -lm ccode.c -fPIC -shared -Wall

in my working directory and in Julia I wrote:

ccode.jl

p=zeros(Float64,1000,3000)
a=rand(Float64,100,1000)
b=rand(Float64,3000,100)

Base.cconvert(::Type{Ptr{Ptr{Cdouble}}},matrix::Matrix{Float64})=Ref{Ptr{Cdouble}}([Ref(matrix,i) for i=1:size(matrix,1):length(matrix)])

function tryccall(p,a,b,nItems,nStud)
#here some tries to convert the matrix in 2d array of pointers:
#p = [Ref(p,i) for i=1:size(p,1):length(p)]
#a = [Ref(a,i) for i=1:size(a,1):length(a)]
#b  = [Ref(b,i) for i=1:size(b,1):length(b)]
#p=pointer(p)
#a=pointer(a)
#b=pointer(b)
#p=Base.cconvert(Ptr{Ptr{Cdouble}},p)
#b=Base.cconvert(Ptr{Ptr{Cdouble}},b)
#a=Base.cconvert(Ptr{Ptr{Cdouble}},a)

ccall(("prob", "$pwd()//ccode"), Cvoid, (Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Cint, Cint), p, a, b, 1000,3000)
#eval(:(ccall((:prob, "$pwd()//ccode"), Cvoid, (Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Cint, Cint), p, pars, latents, 1000,3000)))
end

But I cannot load the library ccode.so. I also tried to add it with Libdl.opendl but it still doesn’t work.
Also adding the working directory to the LOAD_PATH path doesn’t work.

also I get the following error:
TypeError: in ccall: first argument not a pointer or valid constant expression, expected Ptr, got Tuple{String,String}

I tried to use a symbol but it’s still the same.
What can I do?

Thank you in advance
Giada

I first open the dynamic library:

_H2LIB = dlopen(joinpath(_H2LIBDIR, "libh2") * _H2LIBNUMBERS * ".so")

and then I create a library function as

function new_hmatrix(rows::T, cols::T) where {T}
    # new_amatrix (uint rows, uint cols)
    return PHmatrix(ccall(dlsym(H2Libjl._H2LIB, :new_hmatrix), 
        _phmatrix, 
        (Cuint, Cuint), 
        rows, cols))
end 

so that as you can see I use dlsym (the C library H2Libjl._H2LIB contains a function called new_hmatrix).

Would that solve your problem?

Unfortunately not, when I try to use the dlopen("workingdir/ccode.so") function it says:

ERROR: could not load library "workingdir/ccode.so"
The specified module could not be found.

I tried also with “\\” in the path using joinpath but still doesn’t work.

A few things:

  • You want to compile this as a c-library so use gcc instead of g++.
  • Second, the path name needs to be a constant so pull it out as const lib = "..."
  • Thirdly, this is probably not what you want.
    julia> "$pwd()//ccode"
    "pwd()//ccode"
    

There are some more issues but that should at least get you to start segfaulting :slight_smile:

  1. thanks, I changed it to gcc.
  2. I tried both const lib="fullpathtoworkingdir//ccode.so" and const lib="ccode.so", also without “.so”.
  3. You are right again, but it doesn’t work anyway with any of the possible paths!

I restarted Julia and tried to push the working directory path in the Libdl.DL_LOAD_PATH, if I do

const lib="ccode.so"
dlopen(lib)

now I have

could not load library "ccode"
The operation completed successfully.

instead of

could not load library "ccode"
The specified module could not be found.

So, I don’t think I’ve done anything useful :frowning:

# The setup
import Libdl
Libdl

const ccode_lib = joinpath(@__DIR__,"ccode.so")
Libdl.dlopen(ccode_lib)

# The code
p = zeros(Float64,1000,3000)
a = rand(Float64,100,1000)
b = rand(Float64,3000,100)

Base.cconvert(::Type{Ptr{Ptr{Cdouble}}},matrix::Matrix{Float64})=Ref{Ptr{Cdouble}}([Ref(matrix,i) for i=1:size(matrix,1):length(matrix)])

ccall(
    ("prob", ccode_lib),
    Cvoid,
    (Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Cint, Cint),
    p, a, b, 1000,3000
)

This should at least get you to

signal (11): Segmentation fault

Looks like it has to be absolute path and const for dlopen and ccall?

The folder looks like:

|-ccode.c
|-ccode.jl
|-ccode.so

and I used this to build the so:

gcc -o ccode.so -lm ccode.c -fPIC -shared -Wall

Nope,
still the same errors.

Just to be more precise: I’m using Julia 1.2 on a Windows 10 computer.

So how are you compiling? In mingw?

Yes, MinGW latest release.

Well, I can’t reproduce the problem.

Here is my “latest” MinGW gcc

gcc.exe (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0

Compile with this:

/c/Program\ Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/gcc -o ccode.so -lm ccode.c -fPIC -shared -Wall

Also Julia v1.2.0 on Windows 10. I have no error.

julia> const ccode_lib = joinpath(@__DIR__,"ccode.so")
"C:\\Work\\Julia-Repo\\ccall_example\\ccode.so"

julia> Libdl.dlopen(ccode_lib)
Ptr{Nothing} @0x0000000067e80000

julia> p = zeros(Float64,2,3)
2×3 Array{Float64,2}:
 0.0  0.0  0.0
 0.0  0.0  0.0
julia> a = rand(Float64,100,2)
100×2 Array{Float64,2}:
 0.123991   0.405554
 0.895298   0.652647
 ⋮
 0.0702061  0.685377
 0.567912   0.451844

julia> b = rand(Float64,3,100)
3×100 Array{Float64,2}:
 0.0509991  0.0508403  0.887033  0.384972  0.00537979  …  0.784158  0.0625842  0.91498   0.0987885  0.6987
 0.891435   0.65253    0.568233  0.537573  0.431059       0.13089   0.297716   0.495508  0.654329   0.730187
 0.0243292  0.427209   0.904488  0.265174  0.538062       0.130987  0.650718   0.370341  0.916315   0.826939

julia> Base.cconvert(::Type{Ptr{Ptr{Cdouble}}},matrix::Matrix{Float64})=Ref{Ptr{Cdouble}}([Ref(matrix,i) for i=1:size(matrix,1):length(matrix)])

julia> ccall(
           ("prob", ccode_lib),
               Cvoid,
                   (Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Ptr{Ptr{Cdouble}}, Cint, Cint),
                       p, a, b, 2,3
                       )
julia> p
2×3 Array{Float64,2}:
 0.906484  0.880958  0.893602
 0.911566  0.866668  0.0

Sorry, what do you mean for:

Compile with this:

/c/Program\ Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/gcc -o ccode.so -lm ccode.c -fPIC -shared -Wall

?

it means I have to run it in the folder in which I have “ccode.c”, right?

Anyway I don’t know what’s wrong, my gcc version is gcc (MinGW.org GCC-8.2.0-3) 8.2.0

What does nm ccode.so say? Can you see the symbol (prob) that you expect?

The file ccode.so it’s not readable (at least I don’t know how to to do it) :sweat_smile:

Use the nm program.

This is the result

68085060 b .bss
68085024 b .bss
68085008 b .bss
68085060 b .bss
68085000 b .bss
68085034 b .bss
68085020 b .bss
68085054 b .bss
68085020 b .bss
68088010 d .CRT$XDA
68088014 d .CRT$XDZ
68088000 d .CRT$XLA
68081b74 t .ctors.65535
68082008 d .data
68082000 d .data
68082004 d .data
68082008 d .data
68082004 d .data
6808e000 N .debug_abbrev
6808b000 N .debug_aranges
6808c000 N .debug_info
6808f000 N .debug_line
680840c8 r .eh_frame
68084494 r .eh_frame
6808412c r .eh_frame
68087070 i .idata$4
6808703c i .idata$4
680870b8 i .idata$5
680870b4 i .idata$5
680870d4 i .idata$5
680870d8 i .idata$5
680870e0 i .idata$5
680870b0 i .idata$5
680870ac i .idata$5
680870a8 i .idata$5
680870c4 i .idata$5
680870e4 i .idata$5
680870e8 i .idata$5
680870ec i .idata$5
680870f0 i .idata$5
680870f4 i .idata$5
680870f8 i .idata$5
680870a4 i .idata$5
680870fc i .idata$5
680870c0 i .idata$5
680870a0 i .idata$5
680870cc i .idata$5
680870a0 i .idata$5
680870bc i .idata$5
680870d4 i .idata$5
680870c8 i .idata$5
68087194 i .idata$6
680871ac i .idata$6
680871bc i .idata$6
68087166 i .idata$6
680871ca i .idata$6
6808724a i .idata$6
68087152 i .idata$6
68087240 i .idata$6
680871dc i .idata$6
68087236 i .idata$6
68087142 i .idata$6
6808722e i .idata$6
68087104 i .idata$6
68087224 i .idata$6
6808711c i .idata$6
6808721e i .idata$6
680871ec i .idata$6
68087214 i .idata$6
680871fa i .idata$6
6808720c i .idata$6
68087204 i .idata$6
68087134 i .idata$6
68087178 i .idata$6
68083048 r .rdata
68083000 r .rdata
6808311c r .rdata$zzz
680831cc r .rdata$zzz
68083108 r .rdata$zzz
680831e0 r .rdata$zzz
68081540 t .text
68081000 t .text
68081b60 t .text
68081790 t .text
68081380 t .text
68081390 t .text
68081290 t .text
68081ab0 t .text
680811b0 t .text
68081440 t .text
68081b60 t .text.startup
68089000 d .tls$AAA
6808901c d .tls$ZZZ
00000000 A .weak.___deregister_frame_info.___EH_FRAME_BEGIN__
00000000 A .weak.___register_frame_info.___EH_FRAME_BEGIN__
680815a0 T ____w64_mingwthr_add_key_dtor
68081630 T ____w64_mingwthr_remove_key_dtor
68088000 D ___crt_xc_end__
68088000 D ___crt_xc_start__
68088000 D ___crt_xi_end__
68088000 D ___crt_xi_start__
68088000 D ___crt_xl_start__
68088010 D ___crt_xp_end__
68088010 D ___crt_xp_start__
68088010 D ___crt_xt_end__
68088010 D ___crt_xt_start__
68081b70 T ___CTOR_LIST__
         w ___deregister_frame_info
68081af8 T ___dllonexit
680813d0 T ___do_global_ctors
68081390 T ___do_global_dtors
68081b7c T ___DTOR_LIST__
68081490 T ___dyn_tls_init@12
68083044 R ___dyn_tls_init_callback
680840c8 R ___EH_FRAME_BEGIN__
68084494 r ___FRAME_END__
68081260 T ___gcc_deregister_frame
680811b0 T ___gcc_register_frame
68080000 A ___ImageBase
68081420 T ___main
680816d0 T ___mingw_TLScallback
         w ___register_frame_info
680831f4 R ___RUNTIME_PSEUDO_RELOC_LIST__
680831f4 R ___RUNTIME_PSEUDO_RELOC_LIST_END__
68081530 T ___tlregdtor
68089020 D ___tls_end__
68089000 D ___tls_start__
68088000 D ___xl_a
68088004 D ___xl_c
68088008 D ___xl_d
6808800c D ___xl_z
68085064 B __bss_end__
68085000 B __bss_start__
6808505c B __CRT_MT
68081b70 T __CTOR_LIST__
68082008 D __data_end__
68082000 D __data_start__
00000000 A __dll__
00000000 A __dll_characteristics__
68081b7c T __DTOR_LIST__
         U __end__
68081af0 T __errno
00000200 A __file_alignment__
68087000 I __head_libkernel32_a
68087014 I __head_libmsvcrt_a
68087104 I __IAT_end__
680870a0 I __IAT_start__
68080000 A __image_base__
680870d4 I __imp____dllonexit
680870d8 I __imp___errno
680870dc I __imp___iob
680870e0 I __imp__abort
680870e4 I __imp__calloc
680870a0 I __imp__DeleteCriticalSection@4
680870a4 I __imp__EnterCriticalSection@4
680870e8 I __imp__exp
680870ec I __imp__fflush
680870f0 I __imp__free
680870a8 I __imp__FreeLibrary@4
680870f4 I __imp__fwrite
680870ac I __imp__GetLastError@0
680870b0 I __imp__GetModuleHandleA@4
680870b4 I __imp__GetProcAddress@8
680870b8 I __imp__InitializeCriticalSection@4
680870bc I __imp__LeaveCriticalSection@4
680870c0 I __imp__LoadLibraryA@4
680870f8 I __imp__malloc
680870c4 I __imp__TlsGetValue@4
680870fc I __imp__vfprintf
680870c8 I __imp__VirtualProtect@16
680870cc I __imp__VirtualQuery@12
68087288 I __libkernel32_a_iname
680872c4 I __libmsvcrt_a_iname
00000000 A __loader_flags__
00000001 A __major_image_version__
00000004 A __major_os_version__
00000004 A __major_subsystem_version__
00000000 A __minor_image_version__
00000000 A __minor_os_version__
00000000 A __minor_subsystem_version__
68081180 T __onexit
680818d0 T __pei386_runtime_relocator
680831f4 R __rt_psrelocs_end
00000000 A __rt_psrelocs_size
680831f4 R __rt_psrelocs_start
680831f4 R __RUNTIME_PSEUDO_RELOC_LIST__
680831f4 R __RUNTIME_PSEUDO_RELOC_LIST_END__
00001000 A __section_alignment__
00001000 A __size_of_heap_commit__
00100000 A __size_of_heap_reserve__
00001000 A __size_of_stack_commit__
00200000 A __size_of_stack_reserve__
00000003 A __subsystem__
6808901c D __tls_end
68085030 B __tls_index
68089000 D __tls_start
68089004 D __tls_used
68081ae8 T _abort
68081150 T _atexit
68081ae0 T _calloc
68081b58 T _DeleteCriticalSection@4
68082000 d _deregister_frame_fn
68081380 T _DllMain@12
68081060 T _DllMainCRTStartup@12
68081b50 T _EnterCriticalSection@4
68081ad8 T _exp
68081ad0 T _fflush
68081ac8 T _free
68081b48 T _FreeLibrary@4
68081ac0 T _fwrite
68081b40 T _GetLastError@0
68081b38 T _GetModuleHandleA@4
68081b30 T _GetProcAddress@8
68085060 B _hmod_libgcc
68081b28 T _InitializeCriticalSection@4
68081b20 T _LeaveCriticalSection@4
68081b18 T _LoadLibraryA@4
68081ab8 T _malloc
6808502c B _mingw_initltsdrot_force
68085028 B _mingw_initltsdyn_force
68085024 B _mingw_initltssuo_force
68085008 b _obj
68081290 T _prob
68081b60 t _register_frame_ctor
68081b10 T _TlsGetValue@4
68081ab0 T _vfprintf
68081b08 T _VirtualProtect@16
68081b00 T _VirtualQuery@12```
68081290 T _prob

Now you know you have an actual dyn library with your function in it. Have you tried

const ccode_lib = joinpath(@__DIR__,"ccode.so")
Libdl.dlopen(ccode_lib)

?

Yes,

ERROR: could not load library "C:\Users\wd\ccode.so"
The specified module could not be found.

Stacktrace:
 [1] #dlopen#3(::Bool, ::typeof(dlopen), ::String, ::UInt32) at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Libdl\src\Libdl.jl:109
 [2] dlopen at C:\cygwin\home\Administrator\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.2\Libdl\src\Libdl.jl:109 [inlined] (repeats 2 times)
 [3] top-level scope at REPL[8]:1

What is your Julia version?

Julia 1.2

The rest? Do versioninfo() please.