How to make a C function compiled by myself available to `ccall`?

General Usage
1

I am completely new to C and I just wanted to see if someone can help me with making a C function available to ccall. As a MWE I have

// example.c file
#include <stddef.h>

void radix_sort(char **strings) {
	const size_t len = sizeof(strings) / sizeof(char *);
	strings[0] = strings[len];
	return;
}

So I wanted to make radix_sort available for ccall. So these are the steps I have followed

  1. Compile the above using gcc example.c -shared -fPIC -o d:/c_lib/example.so
  2. Set the environment variable LD_LIBRARY_PATH to d:/c_lib/. I tried to follow this
  3. In Julia I tried hello = ccall((:radix_sort,"example"), Void, (Vector{String},),["b","a"])

But it gave me an error

ERROR: error compiling anonymous: could not load library "example"
The specified module could not be found.

So it seems that Julia doesn’t know where the library is. So I also tried to compile it like this

gcc -fPIC -c example.c 
gcc -shared d:/c_lib/example.so example.o

and it gave me the same error.

Please help guide me on how to do this. I am on Windows 10 64bit. It seems something I don’t undertand about where Julia is looking for the shared library; it may have to do with my total lack of experience with C given this is the first time I am writing in C.

Background:
I am trying to make a faster string sorting algorithm in Julia but I wanted to benchmark it against a ccall function

2

I have no idea how the library paths are handled in windows, but I am pretty sure Julia try to find the libraries in those globally defined paths.

According to the docs from https://docs.julialang.org/en/stable/manual/calling-c-and-fortran-code/:

Shared libraries and functions are referenced by a tuple of the form (:function, “library”) or (“function”, “library”) where function is the C-exported function name. library refers to the shared library name: shared libraries available in the (platform-specific) load path will be resolved by name, and if necessary a direct path may be specified.

In macOS you just append the path to DYLD_LIBRARY_PATH or LD_LIBRARY_PATH, in Linux to LD_LIBRARY_PATH. So there must be a way to manage these directories under windows.

Also notice that you can specify the direct path, as mentioned in the quote above.

1 Like
3

So the steps 1-2 are correct? I tried

hello = ccall((:radix_sort,"d:\\c_lib\\example.so"), Void, (Vector{String},),["b","a"])

as well and that gave me the same error.

4

Try push!(Libdl.DL_LOAD_PATH, "yourpath") (which unfortunately does not seem to be documented). Or in a pinch put the DLL in the same path as julia.exe.

1 Like
5

I think on Windows you need to call the library “example.dll” rather than “example.so”. Also I’m not sure how Julia handles Windows paths, so may also try forward slashes instead of backward. Thus I’d try these:

hello = ccall((:radix_sort,"d:\\c_lib\\example.dll"), Void, (Vector{String},),["b","a"])
hello = ccall((:radix_sort,"d:/c_lib/example.dll"), Void, (Vector{String},),["b","a"])
6

At least now I am getting a different error:

ERROR: error compiling anonymous: could not load library "d:/c_lib/example.dll"
%1 is not a valid Win32 application.

Stacktrace:
 [1] eval(::Module, ::Any) at .\boot.jl:235
7

I think you probably like to see and try examples from these links:1, 2, 3

I think (but have not time to check it) that problem is also in bold → ccall((:radix_sort,“example”), Void, (Vector{String},),[“b”,“a”]) there has to be Ptr{Cstring} or Ptr{UInt8} (see this and look for gethostname)something similarly “c-like” …

8

%1 is not a valid Win32 application

This sounds weird given that you have 64-bit Windows. I’d:

  1. Check that you use 64-bit Julia.
  2. Compile your dll using some native tools (see this question for a starting point) - from a few sources I assume the format generated by GCC may be different from native Windows’ (especially if you are under Cygwin).

Note that this error is about loading the library, not finding a function inside the library.

9

FWIW, I can reproduce the error. The exact code that works in Linux to load .so files doesn’t work in Windows. I didn’t try dll files.

10

I have Julia 64 bit and I installed MingGW. Now I am exploring various options including installing Code::Block and Clion to see if those tools can help

11

win could be 64bit but Julia is probably 32bit? You could try gcc -m32 ... to create 32bit dll…
or download 64bit Julia.

But if you will be able to do correct dll next task would not be easy.

You have to understand internal structure of String and Vector{String}…

Maybe next code could something clarify:

# I create simple Vector of strings
julia> v = ["ab", "cd"]
2-element Array{String,1}:
 "ab"
 "cd"

# address of first string (1)
julia> pointer(v[1])
Ptr{UInt8} @0x00007fb9029c9558

# vector is array of address of strings (UInt64 because 64bit address space in 64bit Julia):
julia> vv = reinterpret(Ptr{UInt64}, pointer(v))
Ptr{UInt64} @0x00007fb90247eff0

# but data does not start at address which you get! 
# next value is same as is in (1) so you need to shift by 8 (8 byte space is used for storing sizeof)
# but I don't think that feature freeze means that this is guaranteed! 
julia> sptr = unsafe_load(reinterpret(Ptr{UInt64}, vv))+8
0x00007fb9029c9558

# now you could look at string bytes
# 'a' == 0x61
julia> unsafe_load(reinterpret(Ptr{UInt8}, sptr), 1)
0x61

# 'b' == 0x62
julia> unsafe_load(reinterpret(Ptr{UInt8}, sptr), 2)
0x62

# getting next string from vector:
julia> sptr = unsafe_load(vv, 2)+8
0x00007fb9029c9578

# 'c' == 0x63
julia> unsafe_load(reinterpret(Ptr{UInt8}, sptr), 1)
0x63

# 'd' == 0x64
julia> unsafe_load(reinterpret(Ptr{UInt8}, sptr), 2)
0x64

What I am trying to say is that in Julia you have not stored something like simple char**

You have physical address where (in 64bit Julia) you have stored pointers to objects (Julia strings not char* !)
These objects has stored something like char* after 8-th byte.

As I understand - everything about this address semantic is unfreezed/unguaranteed. But I could be wrong. We have probably ask in different topic. Maybe @ScottPJones Str types would have guaranteed binary representation? I don’t know.

So in C you could probably try this:

void radix_sort(char **strings, size_t len) {  /* you need to pass len here */
        /* sizeof(strings) is address size not size of array so next line is mistake*/
	/* const size_t len = sizeof(strings) / sizeof(char *); */ 
        
        /* this would probably work but we are going from zero 
           so we need len-1 for last element */
	strings[0] = strings[len-1];  

        /* this could probably print value of last string from vector */
        printf(string[len-1]+8); 
	return;
}

But please be careful! Everything in c-code I wrote from heart without tests!!! So save everything before executing tests! Crash is highly probable! :stuck_out_tongue:

12

I always install the 64bit Julia. My UInt is UInt64.

I just tried, in 32bit Julia UInt is UInt32. So I have 64bit Julia installed.

13

Seems to be a hard issue. Some guy doing similar things in Python is also running into issues

14

OK. Forget my first two rows, I think what is next still holds.

Are you still stuck on this error: %1 is not a valid Win32 application?

15

Yes. I am stuck on %1 is not a valid Win32 Application

16

I was trying to find with which compiler was your Julia binary compiled. How did you install your Julia? Which version? You could probably look at code and trying to search something like “copyright” or “(c)” if it is not compiled by microsoft compiler.

I think that python is compiled by using msvc but I could be wrong.

17

Just downloaded the Windows 64bit binary from the julialang.org website.

18

I am using MIngGW on Windows (not Cygwin).

19

self extracting exe… :frowning: I am afraid it could be a little more complicated for me (on linux) to extract real julia.exe (or whatever it is named)…

20

It seems to be mingw. So I would try to recompile dll with or without debug…

Edit: Now I found something which could (maybe) help. But maybe it is just junk info.

GNU C99 5.4.0 -m64 -mtune=generic -march=x86-64 -ggdb -O2 -std=gnu99 -fdebug-prefix-map=/cygdrive/i/szsz/tmpp/cygwin64/mingw64-x86_64/mingw64-x86_64-runtime-5.0.2-1.noarch/build=/usr/src/debug/mingw64-x86_64-runtime-5.0.2-1 -fdebug-prefix-map=/cygdrive/i/szsz/tmpp/cygwin64/mingw64-x86_64/mingw64-x86_64-runtime-5.0.2-1.noarch/src/mingw-w64/mingw-w64-crt=/usr/src/debug/mingw64-x86_64-runtime-5.0.2-1

I would try to use same options.
-ggdb (for debug) is probably good start to add first! :slight_smile: