# How to handle C dependency in packages

**URL:** <https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643>\
**Category:** New to Julia\
**Tags:** package\
**Created:** [January 21, 2020, 10:27pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643 "2020-01-21T22:27:01Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [January 21, 2020, 10:27pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/1 "2020-01-21T22:27:01Z")

</div>

Hi,

I am wondering that if I have some C codes in my own package, how should I get the C functions compiled when importing the package? Should I just ship the compiled dynamic libraries \*.so? Are there any examples or good practices to follow?

Thanks!

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 21, 2020, 11:34pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/2 "2020-01-21T23:34:24Z")

</div>

Never did that myself, but FFTW.jl is working this way. See also BinaryBuilder.jl, it seems this package handles binary deps by downloading tarballs registered in this repo: [Yggdrasil](https://github.com/JuliaPackaging/Yggdrasil)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 22, 2020, 12:02am UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/3 "2020-01-22T00:02:43Z")

</div>

Feel free to ask questions if you have any doubt.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [January 22, 2020, 12:37am UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/4 "2020-01-22T00:37:27Z")

</div>

Adding a binary builder to Yggdrasil is the recommended approach. It should be pretty simple for a basic C library. It might be good if someone who’s deep in the BinaryBuilder stuff wrote a tutorial on how to do this for a simple C library.

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 22, 2020, 11:41am UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/5 "2020-01-22T11:41:40Z")

</div>

I’m not deep in the BinaryBuilder.jl but if no one else is going to pickup this task, I can write such tutorial. I think I have a nice example on my mind.

---

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [January 22, 2020, 2:47pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/6 "2020-01-22T14:47:55Z")

</div>

That would be extremely helpful. I briefly went through BinaryBuilder and Yggdrasil, but I still don’t have a clear picture in mind. It seems that docker is required both on Linux and MacOS right?

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 22, 2020, 3:07pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/7 "2020-01-22T15:07:27Z")

</div>

> [@henry2004y](#):
>
> It seems that docker is required both on Linux and MacOS right?

On Linux Docker is not required, UserNS is a better option for that operating system, and the default one.

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 28, 2020, 6:06pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/8 "2020-01-28T18:06:35Z")

</div>

> [@henry2004y](#):
>
> That would be extremely helpful. I briefly went through BinaryBuilder and Yggdrasil, but I still don’t have a clear picture in mind

Hi, with a lot of help from @giordano I prepared a really simple example how to add such C dependency. There is a step by step instruction how to generate JLL module and then reference it from the other module that exposes functionality by wrapping `ccall`.  
This tutorial needs improvement but I think it may be useful at the current stage.

> **[GitHub - jakubwro/SineWaves.jl: Tutorial and showcase of using a binary...](https://github.com/jakubwro/SineWaves.jl)**
>
> Tutorial and showcase of using a binary dependency in Julia - GitHub - jakubwro/SineWaves.jl: Tutorial and showcase of using a binary dependency in Julia

---

<div class="post-metadata">

**Author:** ![henry2004y](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henry2004y/32/9284_2.png) [@henry2004y](https://discourse.julialang.org/u/henry2004y)\
**Post date:** [January 28, 2020, 9:02pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/9 "2020-01-28T21:02:55Z")

</div>

This is so nice! I thought previously that I may need to include the C source codes in the repository, but here you showed it’s not necessary. Is the compiled \*.so libraries valid across all platforms? Will the performance be better if the code is compiled specifically to the target machine?

Just forgive my ignorance 😅

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 28, 2020, 9:08pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/10 "2020-01-28T21:08:38Z")

</div>

Good questions. BinaryBuilder is quite sophisticated, it will run command you specify in docker container using cross compilers for each selected platforms, so no performance penalty is introduced.  
@giordano committed a great [example Makefile](https://github.com/jakubwro/sinewave/blob/master/Makefile), you may have a look on it, it handles flags and extensions for each supported OS.

EDIT: let’s be clear, there will be performance penalty, because for example some specific instructions of your CPU cannot be used, just general x64 set. It could matter for you if you are Gentoo Linux type of user and you compile your kernel, but in that case you should have your own Julia build too 🙂

---

<div class="post-metadata">

**Author:** ![reumle](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@reumle](https://discourse.julialang.org/u/reumle)\
**Post date:** [January 28, 2020, 9:32pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/11 "2020-01-28T21:32:41Z")

</div>

Your tutorial is a **great** idea! 😃

but it fails on my Windows10 pc. ☹

I get this error:  
`IOError: could not spawn `docker inspect --type=image julia\_binarybuilder\_rootfs:v2020.1.7-5d7e01`: no such file or directory (ENOENT)`

below the full REPL output:

```julia
                        # Step 1: Select your platforms

Make a platform selection
   All Supported Platforms
   Select by Operating System
 > Fully Custom Platform Choice

Select platforms
[press: d=done, a=all, n=none]
^ [] Linux(:armv7l, libc=:glibc, call_abi=:eabihf)
   [] Linux(:powerpc64le, libc=:glibc)
   [] Linux(:i686, libc=:musl)
   [] Linux(:x86_64, libc=:musl)
   [] Linux(:aarch64, libc=:musl)
   [] Linux(:armv7l, libc=:musl, call_abi=:eabihf)
   [] MacOS(:x86_64)
   [] FreeBSD(:x86_64)
   [] Windows(:i686)
 > [X] Windows(:x86_64)

                        # Step 2a: Obtain the source code

Please enter a URL (git repository or compressed archive) containing the source code to build:
> https://github.com/jakubwro/sinewave

The entered URL has been canonicalized to
https://github.com/jakubwro/sinewave.git

Cloning: 100%|█████████████████████████████████████████████████████████████| Time: 0:00:01
Cloning: 100%|█████████████████████████████████████████████████████████████| Time: 0:00:01
Resolving Deltas: 100%|█████████████████████████████████████████████████████████████| Time: 0:00:01You have selected a git repository. Please enter a branch, commit or tag to use.
Please note that for reproducability, the exact commit will be recorded,
so updates to the remote resource will not be used automatically;
you will have to manually update the recorded commit.
> master

Recorded as 609dbc803095d577c83478c5ef8e962613fb16a6

Would you like to download additional sources? [y/N]: N

                        # Step 2b: Obtain binary dependencies (if any)

Do you require any (binary) dependencies? [y/N]: N

[ Info: Updating bare Yggdrasil clone in deps/Yggdrasil...
Enter a name for this project. This will be used for filenames:
> sinewave

Enter a version number for this project:
> 0.1.0

                        # Step 3: Build for Windows(:x86_64)

You will now be dropped into the cross-compilation environment.
Please compile the package. Your initial compilation target is x86_64-w64-mingw32
The $prefix environment variable contains the target directory.
Once you are done, exit by typing `exit` or `^D`

You have the following contents in your working directory:
  - sinewave

IOError: could not spawn `docker inspect --type=image julia_binarybuilder_rootfs:v2020.1.7-5d7e01`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] setup_stdios(::Base.var"#554#555"{Cmd}, ::Array{Any,1}) at .\process.jl:112
 [3] success(::Cmd) at .\process.jl:111
 [4] #import_docker_image#279(::Bool, ::typeof(BinaryBuilder.import_docker_image), ::BinaryBuilder.CompilerShard, ::String) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\DockerRunner.jl:42 [5] #import_docker_image at .\array.jl:0 [inlined]
 [6] #DockerRunner#280(::String, ::Windows, ::Array{Pair{String,String},1}, ::Dict{String,String},
::Bool, ::String, ::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type{BinaryBuilder.DockerRunner}, ::String) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\DockerRunner.jl:104
 [7] (::Core.var"#kw#Type")(::NamedTuple{(:cwd, :workspaces, :platform, :src_name),Tuple{String,Array{Pair{String,String},1},Windows,String}}, ::Type{BinaryBuilder.DockerRunner}, ::String) at .\none:0
 [8] step34(::BinaryBuilder.WizardState) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\wizard\interactive_build.jl:304
 [9] run_wizard(::Nothing) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\Wizard.jl:92
 [10] run_wizard() at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\Wizard.jl:75
 [11] top-level scope at none:0
 [12] eval at .\boot.jl:330 [inlined]
 [13] repleval(::Module, ::Expr) at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:149
 [14] (::Atom.var"#172#174"{Module})() at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:171
 [15] with_logstate(::Atom.var"#172#174"{Module}, ::Base.CoreLogging.LogState) at .\logging.jl:395
 [16] with_logger at .\logging.jl:491 [inlined]
 [17] evalrepl(::Module, ::String) at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:162
 [18] top-level scope at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:207
 [19] eval(::Module, ::Any) at .\boot.jl:330
 [20] eval_user_input(::Any, ::REPL.REPLBackend) at C:\desktop\Julia-1.3.0\share\julia\stdlib\v1.3\REPL\src\REPL.jl:86
 [21] macro expansion at C:\desktop\Julia-1.3.0\share\julia\stdlib\v1.3\REPL\src\REPL.jl:118 [inlined]
 [22] (::REPL.var"#26#27"{REPL.REPLBackend})() at .\task.jl:333

WizardState [step3]

```

Any advice gladly welcome!

[in case it helps]

````julia
julia> versioninfo()
Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = "C:\Users\elm\AppData\Local\atom\app-1.43.0\atom.exe" -a
  JULIA_HOME = C:\desktop\Julia-1.3.0\bin
  JULIA_NUM_THREADS = 3```
````

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 28, 2020, 9:33pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/12 "2020-01-28T21:33:05Z")

</div>

> [@henry2004y](#):
>
> Is the compiled \*.so libraries valid across all platforms?

You don’t get a single library for all platforms, but a library for each platform. To be more precise, a tarball with all libraries and executables, see for example the release [`v0.1.0+0`](https://github.com/jakubwro/sinewave_jll/releases/tag/sinewave-v0.1.0%2B0) of `sinewave_jll`.

> [@henry2004y](#):
>
> Will the performance be better if the code is compiled specifically to the target machine?

I compared a couple of computationally intensive libraries, by natively building them locally and using the one produced by BinaryBuilder, could not see any sensible performance difference.

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 28, 2020, 9:35pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/13 "2020-01-28T21:35:36Z")

</div>

Yes, I was not testing on Windows, and as far as I know BinaryBuilder is not “windows ready” yet. Anyway you can crosscompile windows dlls on mac or linux and use them on Windows.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 28, 2020, 9:38pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/14 "2020-01-28T21:38:40Z")

</div>

Yes, as Jakub said BinaryBuilder unfortunately doesn’t support Windows yet, see for example [https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/520](https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/520).

However, if you manage to write down a tentative builder script using as template one of the several builders in [Yggdrasil](https://github.com/JuliaPackaging/Yggdrasil/) and you then submit a PR to that repository we can have a look there.

---

<div class="post-metadata">

**Author:** ![reumle](https://avatars.discourse-cdn.com/v4/letter/r/5daacb/32.png) [@reumle](https://discourse.julialang.org/u/reumle)\
**Post date:** [January 28, 2020, 10:46pm UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/15 "2020-01-28T22:46:01Z")

</div>

Thanks for the offer! I will do this!

A bit of context :  
For me, compiling `scs` is a stepping stone to the compilation of `superScs` , another optimisation Library ('m not making this up… ), who claims (plausibly) to be better than `scs`, and is also very similar in terms of code structure. (This means SCS.jl as a wrapper might just work) …superSCS is not available to Julia users, but to Matlab and python at cvxgrp, a cousin (uncle?) of convex.jl

- See this for reference: [https://github.com/cvxgrp/cvxpy/pull/565](https://github.com/cvxgrp/cvxpy/pull/565)
- [https://kul-forbes.github.io/scs/](https://kul-forbes.github.io/scs/) for some text (this is superScs)
- [https://github.com/kul-forbes/scs](https://github.com/kul-forbes/scs) for the code
- the article.

```julia
A. Themelis and P. Patrinos, 
"SuperMann: a superlinearly convergent algorithm for finding fixed points of nonexpansive operators," 
arXiv:1609.06955, 2017.

```

---

<div class="post-metadata">

**Author:** ![Jakub\_Wronowski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jakub_wronowski/32/204030_2.png) [@Jakub\_Wronowski](https://discourse.julialang.org/u/Jakub_Wronowski)\
**Post date:** [January 29, 2020, 9:05am UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/16 "2020-01-29T09:05:50Z")

</div>

I tried to build scs with BinaryBuilder and expose JLL package but it tries to link with `-lblas -llapack`. Is there any JLL that contains those libs?

```julia
sandbox:${WORKSPACE}/srcdir/scs # make out/libscsdir.so
mkdir -p out/obj
cc -g -Wall -std=c99 -Wwrite-strings -funroll-loops -Wstrict-prototypes -I. -Iinclude -fPIC -Ofast -DSVD_ACTIVATED=1 -DCTRLC=1 -DCOPYAMATRIX=1 -DLAPACK_LIB_FOUND -DUSE_LAPACK -shared -Wl,-soname,libscsdir.so -o out/libscsdir.so out/obj/scs.o out/obj/util.o out/obj/cones.o out/obj/cs.o out/obj/linAlg.o out/obj/ctrlc.o out/obj/scs_version.o out/obj/directions.o out/obj/unit_test_util.o out/obj/scs_parser.o linsys/direct/private.o linsys/direct/external/ldl.o linsys/direct/external/amd_control.o linsys/direct/external/amd_defaults.o linsys/direct/external/amd_global.o linsys/direct/external/amd_info.o linsys/direct/external/amd_aat.o linsys/direct/external/amd_preprocess.o linsys/direct/external/amd_order.o linsys/direct/external/amd_post_tree.o linsys/direct/external/amd_postorder.o linsys/direct/external/amd_2.o linsys/direct/external/amd_valid.o linsys/direct/external/amd_1.o linsys/common.o -lm -lrt -lblas -llapack
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -lblas
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -llapack
collect2: error: ld returned 1 exit status
make: *** [Makefile:82: out/libscsdir.so] Error 1

```

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 29, 2020, 9:30am UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/17 "2020-01-29T09:30:13Z")

</div>

`OpenBLAS_jll` should work, but Julia’s OpenBLAS is very tricky to handle, I need to write some documentation specifically for that package.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 29, 2020, 9:37am UTC](https://discourse.julialang.org/t/how-to-handle-c-dependency-in-packages/33643/18 "2020-01-29T09:37:06Z")

</div>

It turns out there is already a builder for scs: [https://github.com/JuliaOpt/SCSBuilder](https://github.com/JuliaOpt/SCSBuilder). It would be better to move to Yggdrasil, though