# Julia package to wrap a C library

**URL:** <https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749>\
**Category:** General Usage\
**Tags:** package\
**Created:** [July 24, 2019, 3:50pm UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749 "2019-07-24T15:50:11Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![hros](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@hros](https://discourse.julialang.org/u/hros)\
**Post date:** [July 24, 2019, 3:50pm UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/1 "2019-07-24T15:50:11Z")

</div>

I would like to use [xxHash](https://github.com/Cyan4973/xxHash) which has been ported to almost every language ([link](https://cyan4973.github.io/xxHash/#other-languages)).  
The algorithm is straightforward and I could port it (using the Go version as reference).  
However, the original C version is

1. highly optimized
2. adapts to the system architecture
3. may be updated in the future

So I would like to create a package that upon installation clones the current version of the xxHash repository, adds the necessary C files to wrap the functions, compiles everything and exports a Julia wrapper (using `ccall`).  
It would be nice if the package manager would recognize updates to the dependent repository as a Julia package update.

Is there a demo project / template for accomplishing this.

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 24, 2019, 7:35pm UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/2 "2019-07-24T19:35:28Z")

</div>

> [@hros](#):
>
> So I would like to create a package that upon installation clones the current version of the xxHash repository, adds the necessary C files to wrap the functions, compiles everything and exports a Julia wrapper (using `ccall` ).

The most popular approach in Julia is to use [BinaryBuilder.jl](https://github.com/JuliaPackaging/BinaryBuilder.jl) and [BinaryProvider.jl](https://github.com/JuliaPackaging/BinaryProvider.jl). _Don’t_ build from source on the end-user’s machine, because that requires the user to have a working C compiler (most people don’t on Mac and Windows) and depends on their environment in a fragile way. And you _don’t_ want to always build the latest git master version, as that might break unexpectedly — you want to build a particular git tag known to work, and update to a newer tag periodically (first testing to make sure the newer version still works).

With BinaryBuilder, you create a separate “builder” package (e.g. xxHashBuilder) with a build script to compile the program. They have set things up so that you can run the build script on Travis CI (automated build/test systems integrated with GitHub), and it will cross-compile binary versions of the library for all the platforms Julia supports, and upload these binaries to a release on your GitHub repo.

Then, in your Julia package (e.g. xxHash.jl), you use BinaryProvider (with a build.jl script that was automatically generated by xxHashBuilder) — when a user installs your package, it will automatically download the pre-built binaries.

When you want to update to a newer upstream version, you update your build script to point to the new release, re-run BinaryBuilder by tagging a new release of xxHashBuilder, and then update your Julia package with the new build.jl file.

You can find various examples of this linked to from the BinaryBuilder documentation. One pretty simple example that I just created is the [Xsum.jl](https://github.com/stevengj/Xsum.jl) package, which wraps Radford Neal’s [xsum library](https://gitlab.com/radfordneal/xsum), building binaries via an [xsumBuilder package](https://github.com/stevengj/xsumBuilder).

---

<div class="post-metadata">

**Author:** ![jonathanBieler](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jonathanBieler](https://discourse.julialang.org/u/jonathanBieler)\
**Post date:** [July 25, 2019, 9:03am UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/3 "2019-07-25T09:03:48Z")

</div>

On the Julia side you can use [Clang.jl](https://github.com/JuliaInterop/Clang.jl) to automatically generate bindings from C header files. It might be overkill though since your library is quite simple.

---

<div class="post-metadata">

**Author:** ![hros](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@hros](https://discourse.julialang.org/u/hros)\
**Post date:** [July 29, 2019, 7:18am UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/4 "2019-07-29T07:18:54Z")

</div>

Thanks for your informative reply  
I am struggling with the `.travis.yml` setup, specifically the secure api\_key.  
Travis is failing with bad credentials. what is the procedure to generate the encrypted api\_key?  
I have installed the travis command line utility and after `travis login`, I used the result of `travis token` as a parameter to `travis encrypt` and inserted the result in the secure api\_key field of .travis.yml.  
What is the correct way of generating an api key?

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [July 29, 2019, 7:56am UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/5 "2019-07-29T07:56:37Z")

</div>

I recently struggled with this, and for me one problem was that my CI was running on [Travis-ci.com](http://Travis-ci.com) and not [Travis-ci.org](http://Travis-ci.org), so some of the instructions I was following didn’t work. These are the steps I followed in the end:

1. Made a personal access token with the 6 permissions mentioned here [travis-ci setup releases with --github-token - Stack Overflow](https://stackoverflow.com/a/57032190)
2. Run `travis login --pro`
3. Run `echo token | travis encrypt --com -r ericphanson/SDPA_GMP_Builder` where I replaced `token` with the token I got from github (and you should replace the repository with yours)
4. Take the encrypted output, put in the `secure` line of the .travis.yml file (with quotes; not sure if that matters)
5. Make a commit, tag the commit (`git tag v1`), push it (`git push origin v1`)

Then that should start Travis up (Travis also has to be enabled on the repo via the user Settings → Applications menus). I think to use .org version instead of .com, the “pro” and “com” flags are switched for “org”, or something like that.

Hope that helps!

---

<div class="post-metadata">

**Author:** ![hros](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@hros](https://discourse.julialang.org/u/hros)\
**Post date:** [July 29, 2019, 11:48am UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/6 "2019-07-29T11:48:42Z")

</div>

Thanks. the token works.  
Now travis created the necessary release files, and I copied the generated `build.jl` over to the package repository.  
When I try to test building the repo (currently without any functionality, only a bare skeleton module) I did the following:

- in the cloned repo directory I started the REPL
- entered the `] (build) ` mode, activated the local package (`activate .`)
- starting building the package with `build`  
The last command of `build.jl`, namely:

```julia
write_deps_file(joinpath(@ __DIR__ , "deps.jl"), products, verbose=verbose)

```

fails (when this line is commented the build completes).  
This is the log:

```julia
(xxHash) pkg> build
  Building xxHash → `~/Projects/xxHash.jl/deps/build.log`
 Resolving package versions...
┌ Error: Error building `xxHash`: 
│ ERROR: LoadError: LibraryProduct(nothing, ["libxxhash"], :libxxhash, "Prefix(/home/user/Projects/xxHash.jl/deps/usr)") is not satisfied, cannot generate deps.jl!
│ Stacktrace:
│ [1] error(::String) at ./error.jl:33
│ [2] #write_deps_file#152(::Bool, ::Function, ::String, ::Array{LibraryProduct,1}) at /home/user/.julia/packages/BinaryProvider/TcAwt/src/Products.jl:414
│ [3] (::getfield(BinaryProvider, Symbol("#kw##write_deps_file")))(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(write_deps_file), ::String, ::Array{LibraryProduct,1}) at ./none:0
│ [4] top-level scope at none:0
│ [5] include at ./boot.jl:326 [inlined]
│ [6] include_relative(::Module, ::String) at ./loading.jl:1038
│ [7] include(::Module, ::String) at ./sysimg.jl:29
│ [8] include(::String) at ./client.jl:403
│ [9] top-level scope at none:0
│ in expression starting at /home/user/Projects/xxHash.jl/deps/build.jl:48
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Pkg/src/Operations.jl:1075

```

---

<div class="post-metadata">

**Author:** ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)\
**Post date:** [July 29, 2019, 12:29pm UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/7 "2019-07-29T12:29:28Z")

</div>

> [@hros](#):
>
> `LoadError: LibraryProduct(nothing, ["libxxhash"],`

Try downloading the `.tar.gz` file for your platform manually. Unpack it to make sure it contains `libxxhash.so` (or `.dylib` or `.dll`, depending on your platform). Try doing `import Libdl; Libdl.dlopen("...")` on the shared library to make sure it opens without error.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [July 29, 2019, 12:31pm UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/8 "2019-07-29T12:31:06Z")

</div>

I just had a quick look at your github repo; I think after `make` you need to copy the file `libxxhash` to `${prefix}/lib` (and possibly create the `lib` directory first). My understanding is BinaryBuilder looks in `$prefix` for the output of your build, and it’s not finding it currently.

---

<div class="post-metadata">

**Author:** ![thofma](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thofma/32/1691_2.png) [@thofma](https://discourse.julialang.org/u/thofma)\
**Post date:** [July 29, 2019, 1:18pm UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/9 "2019-07-29T13:18:26Z")

</div>

Usually the wizzard should complain about this, or at least give a warning.

A `make install` with the correct prefix should be sufficient and more robust.

---

<div class="post-metadata">

**Author:** ![hros](https://avatars.discourse-cdn.com/v4/letter/h/97f17d/32.png) [@hros](https://discourse.julialang.org/u/hros)\
**Post date:** [July 31, 2019, 9:09am UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/10 "2019-07-31T09:09:27Z")

</div>

Thanks for all your help, the package is working, and will be published soon.  
I do have one minor question about the build script used in `build_tarballs`. Following @stevengj 's [xsumBuilder](https://github.com/stevengj/xsumBuilder) package I used the `dlext` environment variable in my build script to copy the library to the `lib` sub-directory.  
However, the windows-ming32 target failed (probably because `dlext=dll`), so my current version does not create a mingw32 archive.

```julia
script = raw"""
cd $WORKSPACE/srcdir/xxHash-0.7.0
mkdir -p ${prefix}/lib
make
if ["$dlext" == "dylib"] || ["$dlext" == "so"]
then
    cp libxxhash.${dlext} ${prefix}/lib/libxxhash.${dlext}
fi
"""

```

I checked the mingw32 log and it creates a shared library with the `so` suffix. What environment variable should be used to detect the mingw32 platform?

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [July 31, 2019, 10:31am UTC](https://discourse.julialang.org/t/julia-package-to-wrap-a-c-library/26749/11 "2019-07-31T10:31:27Z")

</div>

Glad it’s working! I am no expert, but I came across this example in the binary builder docs ([here](https://juliapackaging.github.io/BinaryBuilder.jl/latest/build_tips.html#Initiating-different-shell-commands-based-on-target-1)): [https://github.com/davidanthoff/ReadStatBuilder/blob/cc1745add155224ef1672e7a0013c4adb1df8141/build\_tarballs.jl#L33](https://github.com/davidanthoff/ReadStatBuilder/blob/cc1745add155224ef1672e7a0013c4adb1df8141/build_tarballs.jl#L33). Does that do the check you’re looking for?