Julia package to wrap a C library

General Usage
1

I would like to use xxHash which has been ported to almost every language (link).
The algorithm is straightforward and I could port it (using the Go version as reference).
However, the original C version is

  1. highly optimized
  2. adapts to the system architecture
  3. may be updated in the future

So I would like to create a package that upon installation clones the current version of the xxHash repository, adds the necessary C files to wrap the functions, compiles everything and exports a Julia wrapper (using ccall).
It would be nice if the package manager would recognize updates to the dependent repository as a Julia package update.

Is there a demo project / template for accomplishing this.

2

The most popular approach in Julia is to use BinaryBuilder.jl and BinaryProvider.jl. Don’t build from source on the end-user’s machine, because that requires the user to have a working C compiler (most people don’t on Mac and Windows) and depends on their environment in a fragile way. And you don’t want to always build the latest git master version, as that might break unexpectedly — you want to build a particular git tag known to work, and update to a newer tag periodically (first testing to make sure the newer version still works).

With BinaryBuilder, you create a separate “builder” package (e.g. xxHashBuilder) with a build script to compile the program. They have set things up so that you can run the build script on Travis CI (automated build/test systems integrated with GitHub), and it will cross-compile binary versions of the library for all the platforms Julia supports, and upload these binaries to a release on your GitHub repo.

Then, in your Julia package (e.g. xxHash.jl), you use BinaryProvider (with a build.jl script that was automatically generated by xxHashBuilder) — when a user installs your package, it will automatically download the pre-built binaries.

When you want to update to a newer upstream version, you update your build script to point to the new release, re-run BinaryBuilder by tagging a new release of xxHashBuilder, and then update your Julia package with the new build.jl file.

You can find various examples of this linked to from the BinaryBuilder documentation. One pretty simple example that I just created is the Xsum.jl package, which wraps Radford Neal’s xsum library, building binaries via an xsumBuilder package.

15 Likes
3

On the Julia side you can use Clang.jl to automatically generate bindings from C header files. It might be overkill though since your library is quite simple.

1 Like
4

Thanks for your informative reply
I am struggling with the .travis.yml setup, specifically the secure api_key.
Travis is failing with bad credentials. what is the procedure to generate the encrypted api_key?
I have installed the travis command line utility and after travis login, I used the result of travis token as a parameter to travis encrypt and inserted the result in the secure api_key field of .travis.yml.
What is the correct way of generating an api key?

5

I recently struggled with this, and for me one problem was that my CI was running on Travis-ci.com and not Travis-ci.org, so some of the instructions I was following didn’t work. These are the steps I followed in the end:

  1. Made a personal access token with the 6 permissions mentioned here travis-ci setup releases with --github-token - Stack Overflow
  2. Run travis login --pro
  3. Run echo token | travis encrypt --com -r ericphanson/SDPA_GMP_Builder where I replaced token with the token I got from github (and you should replace the repository with yours)
  4. Take the encrypted output, put in the secure line of the .travis.yml file (with quotes; not sure if that matters)
  5. Make a commit, tag the commit (git tag v1), push it (git push origin v1)

Then that should start Travis up (Travis also has to be enabled on the repo via the user Settings → Applications menus). I think to use .org version instead of .com, the “pro” and “com” flags are switched for “org”, or something like that.

Hope that helps!

1 Like
6

Thanks. the token works.
Now travis created the necessary release files, and I copied the generated build.jl over to the package repository.
When I try to test building the repo (currently without any functionality, only a bare skeleton module) I did the following:

  • in the cloned repo directory I started the REPL
  • entered the ] (build) mode, activated the local package (activate .)
  • starting building the package with build
    The last command of build.jl, namely:
write_deps_file(joinpath(@__DIR__, "deps.jl"), products, verbose=verbose)

fails (when this line is commented the build completes).
This is the log:

(xxHash) pkg> build
  Building xxHash → `~/Projects/xxHash.jl/deps/build.log`
 Resolving package versions...
┌ Error: Error building `xxHash`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libxxhash"], :libxxhash, "Prefix(/home/user/Projects/xxHash.jl/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/user/.julia/packages/BinaryProvider/TcAwt/src/Products.jl:414
│  [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│  [4] top-level scope at none:0
│  [5] include at ./boot.jl:326 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1038
│  [7] include(::Module, ::String) at ./sysimg.jl:29
│  [8] include(::String) at ./client.jl:403
│  [9] top-level scope at none:0
│ in expression starting at /home/user/Projects/xxHash.jl/deps/build.jl:48
â”” @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075
7

Try downloading the .tar.gz file for your platform manually. Unpack it to make sure it contains libxxhash.so (or .dylib or .dll, depending on your platform). Try doing import Libdl; Libdl.dlopen("...") on the shared library to make sure it opens without error.

1 Like
8

I just had a quick look at your github repo; I think after make you need to copy the file libxxhash to ${prefix}/lib (and possibly create the lib directory first). My understanding is BinaryBuilder looks in $prefix for the output of your build, and it’s not finding it currently.

2 Likes
9

Usually the wizzard should complain about this, or at least give a warning.

A make install with the correct prefix should be sufficient and more robust.

10

Thanks for all your help, the package is working, and will be published soon.
I do have one minor question about the build script used in build_tarballs. Following @stevengj 's xsumBuilder package I used the dlext environment variable in my build script to copy the library to the lib sub-directory.
However, the windows-ming32 target failed (probably because dlext=dll), so my current version does not create a mingw32 archive.

script = raw"""
cd $WORKSPACE/srcdir/xxHash-0.7.0
mkdir -p ${prefix}/lib
make
if [ "$dlext" == "dylib" ] || [ "$dlext" == "so" ]
then
    cp libxxhash.${dlext} ${prefix}/lib/libxxhash.${dlext}
fi
"""

I checked the mingw32 log and it creates a shared library with the so suffix. What environment variable should be used to detect the mingw32 platform?

1 Like
11

Glad it’s working! I am no expert, but I came across this example in the binary builder docs (here): https://github.com/davidanthoff/ReadStatBuilder/blob/cc1745add155224ef1672e7a0013c4adb1df8141/build_tarballs.jl#L33. Does that do the check you’re looking for?

1 Like