# LAPACK and BLAS inside BinaryBuilder's sandboxed environment

**URL:** <https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700>\
**Category:** Tooling\
**Tags:** binarybuilder\
**Created:** [September 30, 2018, 3:21pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700 "2018-09-30T15:21:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [September 30, 2018, 3:21pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/1 "2018-09-30T15:21:42Z")

</div>

I am learning to use BinaryBuilder for making a binary dependency (ElTopo) available on Linux, Win and Mac. On Linux I use a following recipe to produce a shared library `eltopo.so`:

```julia
git clone https://github.com/tysonbrochu/eltopo
cd eltopo/eltopo3d

cat <<EOF > Makefile.local_defs
INCLUDE_PATH = -I. -I../common -I../common/newsparse -I../common/meshes -I../common/tunicate
DEPEND = g++ -D __LITTLE_ENDIAN__ -DUSE_FORTRAN_BLAS -DNO_GUI
CC = g++ -Wall -D __LITTLE_ENDIAN__ -DUSE_FORTRAN_BLAS -DNO_GUI -fPIC 
RELEASE_FLAGS = -O3 -funroll-loops
DEBUG_FLAGS = -g
LINK = g++
LINK_LIBS = -lGL -lGLU -lglut -llapack -lblas 
EOF

make release
make depend

g++ obj/*.o -o eltopo.so -fPIC -shared -llapack -lblas -lstdc++ -lm -I../common -I.

```

But that does not work inside the sandboxed environment of BinaryBuilder complaining that I don’t have LAPACK and BLAS:

```julia
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -llapack
/opt/x86_64-linux-gnu/bin/../lib/gcc/x86_64-linux-gnu/4.8.5/../../../../x86_64-linux-gnu/bin/ld: cannot find -lblas

```

How could I get LAPACK and BLAS inside sandboxed environment?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [September 30, 2018, 5:50pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/2 "2018-09-30T17:50:32Z")

</div>

I think the right way would be to depend on [https://github.com/JuliaLinearAlgebra/OpenBLASBuilder](https://github.com/JuliaLinearAlgebra/OpenBLASBuilder)

IIRC the Wizard has an option for that.

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [October 2, 2018, 8:24am UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/3 "2018-10-02T08:24:26Z")

</div>

Thanks that actually worked! I replaced `-lblas` and `-llapac`with `-L./destdir/lib -lopenblas64_`. But I do have another question. My `build_tarbals.jl` script now looks as follows

```julia
# Note that this script can accept some limited command-line arguments, run
# `julia build_tarballs.jl --help` to see a usage message.
using BinaryBuilder

name = "ElTopoBinary"
version = v"0.1.0"

# Collection of sources required to build ElTopoBinary
sources = [
    "https://github.com/tysonbrochu/eltopo.git" =>
    "14b1d7cbd45def90cfce04d381e92c4fefc5fab7",

]

# Bash recipe for building across all platforms
script = raw"""
cd $WORKSPACE/srcdir

cd eltopo/eltopo3d/

cat <<EOF > Makefile.local_defs
INCLUDE_PATH = -I. -I../common -I../common/newsparse -I../common/meshes -I../common/tunicate
DEPEND = g++ -D __LITTLE_ENDIAN__ -DUSE_FORTRAN_BLAS -DNO_GUI
CC = g++ -Wall -D __LITTLE_ENDIAN__ -DUSE_FORTRAN_BLAS -DNO_GUI -fPIC 
RELEASE_FLAGS = -O3 -funroll-loops
DEBUG_FLAGS = -g
LINK = g++
LINK_LIBS = -lGL -lGLU -lglut -llapack -lblas 
EOF

make depend
make release

cd $WORKSPACE
(cd srcdir/eltopo/eltopo3d && find . -name '*.h' -print | tar --create --files-from -) | (cd $WORKSPACE/destdir/include && tar xvfp -)
(cd srcdir/eltopo/common && find . -name '*.h' -print | tar --create --files-from -) | (cd $WORKSPACE/destdir/include && tar xvfp -)

g++ srcdir/eltopo/eltopo3d/obj/*.o -o destdir/lib/eltopo.so -fPIC -shared -L./destdir/lib -lopenblas64_ -lstdc++ -lm
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = [
    Linux(:x86_64, :glibc)
    #Linux(:x86_64, libc=:glibc)
    # Linux(:aarch64, libc=:glibc),
    # Linux(:powerpc64le, libc=:glibc),
    # Linux(:x86_64, libc=:musl),
    # Linux(:aarch64, libc=:musl)
]

# The products that we will ensure are always built
products(prefix) = [
    LibraryProduct(prefix, "eltopo", :eltopo)
]

# Dependencies that must be installed before this package can be built
dependencies = [
    "https://github.com/JuliaLinearAlgebra/OpenBLASBuilder/releases/download/v0.3.0-2/build_OpenBLAS.v0.3.0.jl"
]

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies)

```

When I execute it `julia build_tarballs.jl` I get an error after the build script had been finished

```julia
ERROR: LoadError: KeyError: key Linux(:x86_64, libc=:glibc) not found
Stacktrace:
 [1] getindex(::Dict{Platform,Tuple{String,String}}, ::Linux) at ./dict.jl:478
 [2] top-level scope at /home/janiserdmanis/.julia/packages/BinaryBuilder/Y2V0i/src/AutoBuild.jl:412
in expression starting at /home/janiserdmanis/BtSync/Projects/Julia/BinaryBuilder/build/build_tarballs.jl:60

```

What does it mean?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [October 2, 2018, 6:25pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/4 "2018-10-02T18:25:13Z")

</div>

@staticfloat might be able to help you here, but which version of BinaryBuilder are you one?  
There recently was a change to the way platforms work [BinaryBuilder and GCC multi-versioning](https://discourse.julialang.org/t/binarybuilder-and-gcc-multi-versioning/15314)

---

<div class="post-metadata">

**Author:** ![staticfloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/staticfloat/32/34_2.png) [@staticfloat](https://discourse.julialang.org/u/staticfloat)\
**Post date:** [October 3, 2018, 4:00am UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/5 "2018-10-03T04:00:44Z")

</div>

I believe this was a bug in BB that was _just_ fixed. Please update your BinaryBuilder checkout and try again.

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [October 3, 2018, 9:27am UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/6 "2018-10-03T09:27:22Z")

</div>

I was on master. I will check it out again later today.

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [October 3, 2018, 11:22pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/7 "2018-10-03T23:22:21Z")

</div>

Thanks! The build script works and produces a product 🙂 I am amazed by the result. Only binaries and my added header files without stuff of openblas. Seems that building and depending on a binary is going to become a fun thing to do :))

---

<div class="post-metadata">

**Author:** ![non-Jedi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/non-jedi/32/3645_2.png) [@non-Jedi](https://discourse.julialang.org/u/non-Jedi)\
**Post date:** [October 4, 2018, 1:21pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/8 "2018-10-04T13:21:09Z")

</div>

Hold on. Am I reading this correctly? If a package depends on BLAS and LAPACK, the recommended way is to package an additional copy of openBLAS rather than just use the BLAS and LAPACK that Julia already depends on? Why is that?

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [October 4, 2018, 3:03pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/9 "2018-10-04T15:03:28Z")

</div>

In the binary there is nothing of openBLAS except a link to a shared library `libopneblas64_.so` which is only present at the building stage. When tarball is built the openBLAS files are removed so there are no additional copies. I expect there is some magic with BinaryProvider which adds soft links to the libraries for the linking when one tries to install it.(But I will find out that soon.) As I understand, there is nothing which would restrict to make soft links to the same shared library or even to the system one by implementing something similar as `update-alternatives` from Debian.

---

<div class="post-metadata">

**Author:** ![Janis\_Erdmanis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/janis_erdmanis/32/10869_2.png) [@Janis\_Erdmanis](https://discourse.julialang.org/u/Janis_Erdmanis)\
**Post date:** [October 14, 2018, 4:24pm UTC](https://discourse.julialang.org/t/lapack-and-blas-inside-binarybuilders-sandboxed-environment/15700/10 "2018-10-14T16:24:03Z")

</div>

After running some more tests with the compiled shared library which depends on openBLAS I found a following error:

```julia
julia: symbol lookup error: /home/janiserdmanis/BtSync/Projects/Julia/ElTopo.jl/deps/usr/lib/eltopo.so: undefined symbol: dsyev_

```

Looking up I found `dsyev_` is some kind of BLAS function so I looked if the compiled library `eltopo.so` is linked to openBLAS library:

```julia
[janiserdmanis][~/BtSync/Projects/Julia/ElTopo.jl/deps/usr/lib] $ ldd eltopo.so 
        linux-vdso.so.1 (0x00007ffd109e8000)
        libgtk3-nocsd.so.0 => /usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0 (0x00007f7c0f096000)
        libopenblas64_.so.0 => /home/janiserdmanis/BtSync/Projects/Julia/ElTopo.jl/deps/usr/lib/./libopenblas64_.so.0 (0x00007f7c0caff000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7c0c771000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7c0c3d3000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7c0c1bb000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7c0bdca000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7c0bbc6000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7c0b9a7000)
        libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f7c0b5c8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7c0f55b000)
        libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f7c0b388000)

```

where I see that linking seems to be fine. Then I looked up if the symbol is present in `libopenblas64_.so.0`:

```julia
[janiserdmanis][~/BtSync/Projects/Julia/ElTopo.jl/deps/usr/lib] $ nm -D libopenblas64_.so.0 | grep dsyev_
0000000001bdd190 T dsyev_2stage_64_
0000000001b5a330 T dsyev_64_
000000000203d4b0 T LAPACKE_dsyev_2stage64_
000000000203d620 T LAPACKE_dsyev_2stage_work64_
000000000203bce0 T LAPACKE_dsyev_work64_

```

which reveals that for BinaryBuilder’s openBLAS there is a suffix `*64_`. Looking up `SundialsBuilder` seems that the recommended way is to patch the code before compiling like with this one. However that patch seems to be limited to a certain symbols. Are there some universal patch for mangling BLAS and LAPACK symbols to get `*64_` suffix?

If I try to link with openBLAS static object `libopenblas64_.a` I get a following error upon execution of problematic code:

```julia
julia: symbol lookup error: /home/janiserdmanis/BtSync/Projects/Julia/ElTopo.jl/deps/usr/lib/eltopo.so: undefined symbol: _gfortran_pow_r8_i8

```

**EDIT** : Adding `-lgfortran` flag fixed the issue above with static linking.

**EDIT2** : After even more testing I got an unexpected error at runtime:

```julia
** On entry to DSYEV Safe minimumPrecisionM parameter number 3 had an illegal value

```