How to handle C dependency in packages

Hi,

I am wondering that if I have some C codes in my own package, how should I get the C functions compiled when importing the package? Should I just ship the compiled dynamic libraries *.so? Are there any examples or good practices to follow?

Thanks!

Never did that myself, but FFTW.jl is working this way. See also BinaryBuilder.jl, it seems this package handles binary deps by downloading tarballs registered in this repo: Yggdrasil

Feel free to ask questions if you have any doubt.

Adding a binary builder to Yggdrasil is the recommended approach. It should be pretty simple for a basic C library. It might be good if someone who’s deep in the BinaryBuilder stuff wrote a tutorial on how to do this for a simple C library.

8 Likes

I’m not deep in the BinaryBuilder.jl but if no one else is going to pickup this task, I can write such tutorial. I think I have a nice example on my mind.

3 Likes

That would be extremely helpful. I briefly went through BinaryBuilder and Yggdrasil, but I still don’t have a clear picture in mind. It seems that docker is required both on Linux and MacOS right?

On Linux Docker is not required, UserNS is a better option for that operating system, and the default one.

1 Like

Hi, with a lot of help from @giordano I prepared a really simple example how to add such C dependency. There is a step by step instruction how to generate JLL module and then reference it from the other module that exposes functionality by wrapping ccall.
This tutorial needs improvement but I think it may be useful at the current stage.

7 Likes

This is so nice! I thought previously that I may need to include the C source codes in the repository, but here you showed it’s not necessary. Is the compiled *.so libraries valid across all platforms? Will the performance be better if the code is compiled specifically to the target machine?

Just forgive my ignorance :sweat_smile:

Good questions. BinaryBuilder is quite sophisticated, it will run command you specify in docker container using cross compilers for each selected platforms, so no performance penalty is introduced.
@giordano committed a great example Makefile, you may have a look on it, it handles flags and extensions for each supported OS.

EDIT: let’s be clear, there will be performance penalty, because for example some specific instructions of your CPU cannot be used, just general x64 set. It could matter for you if you are Gentoo Linux type of user and you compile your kernel, but in that case you should have your own Julia build too :slight_smile:

Your tutorial is a great idea! :smiley:

but it fails on my Windows10 pc. :frowning:

I get this error:
IOError: could not spawn docker inspect --type=image julia_binarybuilder_rootfs:v2020.1.7-5d7e01: no such file or directory (ENOENT)

below the full REPL output:

                        # Step 1: Select your platforms

Make a platform selection
   All Supported Platforms
   Select by Operating System
 > Fully Custom Platform Choice

Select platforms
[press: d=done, a=all, n=none]
^  [ ] Linux(:armv7l, libc=:glibc, call_abi=:eabihf)
   [ ] Linux(:powerpc64le, libc=:glibc)
   [ ] Linux(:i686, libc=:musl)
   [ ] Linux(:x86_64, libc=:musl)
   [ ] Linux(:aarch64, libc=:musl)
   [ ] Linux(:armv7l, libc=:musl, call_abi=:eabihf)
   [ ] MacOS(:x86_64)
   [ ] FreeBSD(:x86_64)
   [ ] Windows(:i686)
 > [X] Windows(:x86_64)

                        # Step 2a: Obtain the source code

Please enter a URL (git repository or compressed archive) containing the source code to build:
> https://github.com/jakubwro/sinewave

The entered URL has been canonicalized to
https://github.com/jakubwro/sinewave.git

Cloning: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| Time: 0:00:01
Cloning: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| Time: 0:00:01
Resolving Deltas: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| Time: 0:00:01You have selected a git repository. Please enter a branch, commit or tag to use.
Please note that for reproducability, the exact commit will be recorded,
so updates to the remote resource will not be used automatically;
you will have to manually update the recorded commit.
> master

Recorded as 609dbc803095d577c83478c5ef8e962613fb16a6

Would you like to download additional sources?  [y/N]: N

                        # Step 2b: Obtain binary dependencies (if any)

Do you require any (binary) dependencies?  [y/N]: N

[ Info: Updating bare Yggdrasil clone in deps/Yggdrasil...
Enter a name for this project.  This will be used for filenames:
> sinewave

Enter a version number for this project:
> 0.1.0

                        # Step 3: Build for Windows(:x86_64)

You will now be dropped into the cross-compilation environment.
Please compile the package. Your initial compilation target is x86_64-w64-mingw32
The $prefix environment variable contains the target directory.
Once you are done, exit by typing `exit` or `^D`

You have the following contents in your working directory:
  - sinewave

IOError: could not spawn `docker inspect --type=image julia_binarybuilder_rootfs:v2020.1.7-5d7e01`: no such file or directory (ENOENT)
Stacktrace:
 [1] _spawn_primitive(::String, ::Cmd, ::Array{Any,1}) at .\process.jl:99
 [2] setup_stdios(::Base.var"#554#555"{Cmd}, ::Array{Any,1}) at .\process.jl:112
 [3] success(::Cmd) at .\process.jl:111
 [4] #import_docker_image#279(::Bool, ::typeof(BinaryBuilder.import_docker_image), ::BinaryBuilder.CompilerShard, ::String) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\DockerRunner.jl:42 [5] #import_docker_image at .\array.jl:0 [inlined]
 [6] #DockerRunner#280(::String, ::Windows, ::Array{Pair{String,String},1}, ::Dict{String,String},
::Bool, ::String, ::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type{BinaryBuilder.DockerRunner}, ::String) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\DockerRunner.jl:104
 [7] (::Core.var"#kw#Type")(::NamedTuple{(:cwd, :workspaces, :platform, :src_name),Tuple{String,Array{Pair{String,String},1},Windows,String}}, ::Type{BinaryBuilder.DockerRunner}, ::String) at .\none:0
 [8] step34(::BinaryBuilder.WizardState) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\wizard\interactive_build.jl:304
 [9] run_wizard(::Nothing) at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\Wizard.jl:92
 [10] run_wizard() at C:\Users\elm\.julia\packages\BinaryBuilder\4sWut\src\Wizard.jl:75
 [11] top-level scope at none:0
 [12] eval at .\boot.jl:330 [inlined]
 [13] repleval(::Module, ::Expr) at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:149
 [14] (::Atom.var"#172#174"{Module})() at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:171
 [15] with_logstate(::Atom.var"#172#174"{Module}, ::Base.CoreLogging.LogState) at .\logging.jl:395
 [16] with_logger at .\logging.jl:491 [inlined]
 [17] evalrepl(::Module, ::String) at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:162
 [18] top-level scope at C:\Users\elm\.julia\packages\Atom\lBERI\src\repl.jl:207
 [19] eval(::Module, ::Any) at .\boot.jl:330
 [20] eval_user_input(::Any, ::REPL.REPLBackend) at C:\desktop\Julia-1.3.0\share\julia\stdlib\v1.3\REPL\src\REPL.jl:86
 [21] macro expansion at C:\desktop\Julia-1.3.0\share\julia\stdlib\v1.3\REPL\src\REPL.jl:118 [inlined]
 [22] (::REPL.var"#26#27"{REPL.REPLBackend})() at .\task.jl:333

WizardState [step3]

Any advice gladly welcome!

[in case it helps]

julia> versioninfo()
Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = "C:\Users\elm\AppData\Local\atom\app-1.43.0\atom.exe"  -a
  JULIA_HOME = C:\desktop\Julia-1.3.0\bin
  JULIA_NUM_THREADS = 3```

You don’t get a single library for all platforms, but a library for each platform. To be more precise, a tarball with all libraries and executables, see for example the release v0.1.0+0 of sinewave_jll.

I compared a couple of computationally intensive libraries, by natively building them locally and using the one produced by BinaryBuilder, could not see any sensible performance difference.

3 Likes

Yes, I was not testing on Windows, and as far as I know BinaryBuilder is not β€œwindows ready” yet. Anyway you can crosscompile windows dlls on mac or linux and use them on Windows.

Yes, as Jakub said BinaryBuilder unfortunately doesn’t support Windows yet, see for example https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/520.

However, if you manage to write down a tentative builder script using as template one of the several builders in Yggdrasil and you then submit a PR to that repository we can have a look there.

Thanks for the offer! I will do this!

A bit of context :
For me, compiling scs is a stepping stone to the compilation of superScs , another optimisation Library ('m not making this up… ), who claims (plausibly) to be better than scs, and is also very similar in terms of code structure. (This means SCS.jl as a wrapper might just work) …superSCS is not available to Julia users, but to Matlab and python at cvxgrp, a cousin (uncle?) of convex.jl

A. Themelis and P. Patrinos, 
"SuperMann: a superlinearly convergent algorithm for finding fixed points of nonexpansive operators," 
arXiv:1609.06955, 2017.
2 Likes

I tried to build scs with BinaryBuilder and expose JLL package but it tries to link with -lblas -llapack. Is there any JLL that contains those libs?

sandbox:${WORKSPACE}/srcdir/scs # make out/libscsdir.so
mkdir -p out/obj
cc -g -Wall -std=c99 -Wwrite-strings -funroll-loops -Wstrict-prototypes -I. -Iinclude -fPIC -Ofast  -DSVD_ACTIVATED=1 -DCTRLC=1  -DCOPYAMATRIX=1  -DLAPACK_LIB_FOUND -DUSE_LAPACK -shared -Wl,-soname,libscsdir.so -o out/libscsdir.so out/obj/scs.o out/obj/util.o out/obj/cones.o out/obj/cs.o out/obj/linAlg.o out/obj/ctrlc.o out/obj/scs_version.o out/obj/directions.o out/obj/unit_test_util.o out/obj/scs_parser.o linsys/direct/private.o linsys/direct/external/ldl.o linsys/direct/external/amd_control.o linsys/direct/external/amd_defaults.o linsys/direct/external/amd_global.o linsys/direct/external/amd_info.o linsys/direct/external/amd_aat.o linsys/direct/external/amd_preprocess.o linsys/direct/external/amd_order.o linsys/direct/external/amd_post_tree.o linsys/direct/external/amd_postorder.o linsys/direct/external/amd_2.o linsys/direct/external/amd_valid.o linsys/direct/external/amd_1.o linsys/common.o -lm -lrt -lblas -llapack
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -lblas
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -llapack
collect2: error: ld returned 1 exit status
make: *** [Makefile:82: out/libscsdir.so] Error 1

OpenBLAS_jll should work, but Julia’s OpenBLAS is very tricky to handle, I need to write some documentation specifically for that package.

1 Like

It turns out there is already a builder for scs: https://github.com/JuliaOpt/SCSBuilder. It would be better to move to Yggdrasil, though

1 Like