Wrapping a C library | Where to start?

General Usage
, , , ,
1

Hi!

I want to use a few functions of a dll file from Julia and I want to wrap them in a package. The dlls I want to use are for talking to some motion controllers (The dlls in question can be found on thorlabs.com. There are also .h files included.

My first problem is to actually write the wrapping module. From the REPL (and also by includeing a file) I can do

julia> using Libdl
julia> dlopen("Thorlabs.MotionControl.KCube.Solenoid.dll")
Ptr{Nothing} @0x00007ff8be9b0000

with the dll file in the working directory. I am also able to call the functions via ccall.

However, if I try to access this dll file in a module (or its tests) I get an error:

could not load library "Thorlabs.MotionControl.KCube.Solenoid.dll"
  The specified module could not be found.
  Stacktrace:
    [1] open(serial_number::String)
      @ KCubeSolenoid C:\Users\Admin\.julia\dev\KCubeSolenoid\src\KCubeSolenoid.jl:9
    [2] macro expansion
      @ show.jl:955 [inlined]
    [3] macro expansion
      @ C:\Users\Admin\.julia\dev\KCubeSolenoid\test\runtests.jl:9 [inlined]
    [4] macro expansion
      @ C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Test\src\Test.jl:1151 [inlined]
    [5] top-level scope
      @ C:\Users\Admin\.julia\dev\KCubeSolenoid\test\runtests.jl:7
    [6] include(fname::String)
      @ Base.MainInclude .\client.jl:444
    [7] top-level scope
      @ none:6
    [8] eval
      @ .\boot.jl:360 [inlined]
    [9] exec_options(opts::Base.JLOptions)
      @ Base .\client.jl:261
   [10] _start()
      @ Base .\client.jl:485

So I guess the dll is in the wrong location? How would I actually go about and structure the wrapping package? Where should the dll file go?

I briefly tried to wrap the library with Clang.jl but it failed parsing the header file. Are there any tutorials on how to write a wrapper for a simple dll library and ship that as a Julia package?

2

If you don’t want to struggle, the argument to dlopen should be an absolute path (or a relative path to an absolute path like @__DIR__). Libdl.find_library might also be useful.

3 Likes
3

I briefly tried to wrap the library with Clang.jl but it failed parsing the header file.

You may also want to look at using CBinding.jl to create the bindings for you. If you are somewhat comfortable with C development against the library, then using CBinding should be very familiar. I have used it to create bindings for just about everything (GLibC, GLib, GTK, Linux headers) with very little effort:

module MyLib
  using CBinding
  c`-I/include/path -L/lib/path -lmylib`
  c"""
    #include <mylib.h>
  """ji
end

A more sophisticated example

3 Likes
4

Hello,

I am new with Julia, and decided to start by porting one of the File IO libraries that I use in my work a lot.
So far I was successfully able to make a _jll package for the library + basic wrapper using CBinding.jl

Is there a more complete examples on wrapping C libraries ? In particular, I need to deal with automatically releasing resources allocated in C code (i.e calling destructor). And passing multidimensional arrays of different types.

5

Unfortunately, there is no simple answer… The design of the C library and the desired design/usage of the Julia package wrapping it all play a part in that, so you will have to find examples from packages across the Julia ecosystem that utilize a C library that seem to fit your situation. Resource management is a good example of the various approaches you will see used:

For short-lifetime objects, something like this:

x = nothing
try
  x = alloc()
  use(x)
finally
  isnothing(x) || free(x)
end

Which can be wrapped with a function:

function with_resource(func::Function)
  x = nothing
  try
    x = alloc()
    func(x)
  finally
    isnothing(x) || free(x)
  end
end
# then used with
with_resource(x -> use(x))
# or
with_resource() do x
  use(x)
end

Or for objects that can be cleaned up at the leisure of the GC, something like:

struct Resource
  x::Ref

  function Resource()
    ret = new(Ref(alloc()))
    finalizer(x -> free(x[]), ret.x)
    return ret
  end
end

For library setup before the API can be called, you might use something like this:

module Example
  x = nothing
  function __init__()
    global x = alloc()
  end
end

There are probably many more options for multi-dimensional arrays, depending on how they are arranged in memory, etc.

6

thank you, these examples are already very useful.

7

Okay… I finally had time to play around with this a little more. Thanks for all the suggestions!

I tried wrapping the library manually, with Clang.jl, and CBinding.jl but in both cases I seem to have trouble parsing the header files.

My case: I have a header Thorlabs.MotionControl.KCube.Solenoid.h. I downloaded that one from the link in the opening post. I now want to write a wrapper for this library.

Let’s start with my manual attempt

The first thing I tried (and failed at :confused:) was wrapping the SC_Open, and SC_Close functions. The relevant part of the header file is this:

	/// <summary> Open the device for communications. </summary>
	/// <param name="serialNo">	The serial no of the device to be connected. </param>
	/// <returns> The error code (see \ref C_DLL_ERRORCODES_page "Error Codes") or zero if successful. </returns>
    /// 		  \include CodeSnippet_connection1.cpp
	/// <seealso cref="SC_Close(char const * serialNo)" />
	KCUBESOLENOID_API short __cdecl SC_Open(char const * serialNo);

	/// <summary> Disconnect and close the device. </summary>
	/// <param name="serialNo">	The serial no of the device to be disconnected. </param>
    /// 		  \include CodeSnippet_connection1.cpp
	/// <seealso cref="SC_Open(char const * serialNo)" />
	KCUBESOLENOID_API void __cdecl SC_Close(char const * serialNo);

My attempt of wrapping these:

libfile = "C:\\Program Files\\Thorlabs\\Kinesis\\Thorlabs.MotionControl.KCube.Solenoid.dll"
@assert isfile(libfile)
solenoid = Libdl.dlopen(libfile)

function open(serialnumber)
    sym = Libdl.dlsym(solenoid, :SC_Open)
    ccall(sym, cdecl, Int16, (Ref{String}, ), serialnumber)
end

function close(serialnumber)
    sym = Libdl.dlsym(solenoid, :SC_Close)
    ccall(sym, cdecl, Cvoid, (Ref{String}, ), serialnumber)
end

serialnumber = Ref{String}("68210439")

begin
    @show open(serialnumber)
    close(serialnumber)
end

This fails with open(serialnumber) returning an error code that the device was not found (although connected and working with other software). I guess the problem is in the input argument types in the ccall function. I am not sure Ref{String} is the correct Julia type for char const * serialNo.

Clang.jl

To me it seemed a good idea to circumvent this smoothness of my brain by automagically wrapping the header file with Clang.jl. I don’t really have an idea what I’m doing here so I created a generator.toml file with the following content

[general]
library_name = "Thorlabs.MotionControl.KCube.Solenoid"
output_file_path = "./Solenoid.jl"
module_name = "Solenoid"
export_symbol_prefixes = ["SC", "FT", "MOT", "TLI"]

and a script

using Clang.Generators

cd(@__DIR__)

include_dir = normpath("C:/Program Files/Thorlabs/Kinesis/")
kinesis_dir = include_dir

options = load_options(joinpath(@__DIR__, "generator.toml"))

args = get_default_args()
push!(args, "-I$include_dir")

headers = [joinpath(kinesis_dir, "Thorlabs.MotionControl.KCube.Solenoid.h")]

ctx = create_context(headers, args, options)

build!(ctx)

which closely follows the tutorial on the Clang.jl GitHub page. However, Clang.jl seems to have problems parsing the headers. create_context does not throw an error but I get these messages:

[ Info: Parsing headers...
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\ia32intrin.h:95:1: error: definition of builtin function '__rdtsc'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\xmmintrin.h:792:1: error: definition of builtin function '_mm_getcsr'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\xmmintrin.h:824:1: error: definition of builtin function '_mm_setcsr'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\xmmintrin.h:1216:1: error: definition of builtin function '_mm_sfence'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\xmmintrin.h:1225:1: error: definition of builtin function '_mm_pause'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\emmintrin.h:1457:1: error: definition of builtin function '_mm_clflush'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\emmintrin.h:1463:1: error: definition of builtin function '_mm_lfence'
C:\Users\MG222\.julia\artifacts\217a1339fc81b7388fc8af3fc8a05bca7b7ced19\lib\gcc\x86_64-w64-mingw32\4.8.5\include\emmintrin.h:1469:1: error: definition of builtin function '_mm_mfence'
C:\Program Files\Thorlabs\Kinesis\Thorlabs.MotionControl.KCube.Solenoid.h:33:8: error: expected identifier or '('
Context(...)

From my research on the web I came to the conclusion (which I do not fully understand) that these headers contain function definitions which are part of the gcc compiler but are not recognized by clang???

So, because the header file is not parsed correctly I only get this output printed in Solenoid.jl:

module Solenoid

using CEnum

const KCUBESOLENOID_API = __declspec(dllimport)

# exports
const PREFIXES = ["SC", "FT", "MOT", "TLI"]
for name in names(@__MODULE__; all=true), prefix in PREFIXES
    if startswith(string(name), prefix)
        @eval export $name
    end
end

end # module

Is there a way to ignore certain include statements or does everything have to be parsed in order to generate Julia bindings?

CBinding.jl

My naïve first try was

using CBinding

libpath = "C:\\Program Files\\Thorlabs\\Kinesis\\Thorlabs.MotionControl.KCube.Solenoid.dll"

c`$([
    "-std=c99",
    "-Wall",
    "-DGO_FAST=1",
    "-L$(libpath)", "-Thorlabs.MotionControl.KCube.Solenoid"
])`

c"""
#include <C:\\Program Files\\Thorlabs\\Kinesis\\Thorlabs.MotionControl.KCube.Solenoid.h>
"""ji

which resulted in

Error: C:\\Program Files\\Thorlabs\\Kinesis\\Thorlabs.MotionControl.KCube.Solenoid.h:14:10: fatal error: 'OaIdl.h' file not found

The OaIdl.h file seems to be part of the Windows SDK. So I installed that and added the path to this specific header file to the compiler context. This continued with numerous other header files which I had to manually include (I guess that’s not the way to do it) and changing the target in the compiler context to "--target=stable-x86_64-pc-windows-msvc". Only to end up with a parsing error that I am not able to resolve:

┌ Error: C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt\corecrt.h:609:32: error: typedef redefinition with different types ('__crt_locale_pointers *' (aka 'struct __crt_locale_pointers *') vs 'struct localeinfo_struct *')
└ @ CBinding C:\Users\MG222\.julia\packages\CBinding\kBUap\src\context.jl:377
┌ Error: C:\\Program Files\\Thorlabs\\Kinesis\\Thorlabs.MotionControl.KCube.Solenoid.h:33:8: error: expected identifier or '('
└ @ CBinding C:\Users\MG222\.julia\packages\CBinding\kBUap\src\context.jl:377
ERROR: LoadError: Errors parsing C code

…

So any suggestions and ideas on how to resolve my problems with any of the three approaches are highly welcome :slight_smile:.