JuliaPro 0.6.1.1 cannot build/use ZMQ

zmq
juliapro

#1

I am trying to install JuliaPro 0.6.1.1 (MKL variant) on an Ubuntu 17.10 computer and the installer fails when it tries to build ZMQ. On the contrary, when I use the standard Julia 0.6.1 I can add and properly build ZMQ.

On Ubuntu 17.10 there is library libzmq.so.5 installed in /usr/lib/x86_64_linux-gnu/ and so when using the standard Julia 0.6.1 REPL the installation of ZMQ detects this library and does not download any source code from the web to compile and use. On the contrary, JuliaPro 0.6.1.1 does not detect this library and it always downloads and compiles source code from the web. The compilation generates no errors but then BinDeps cannot launch this library and therefore the installation is marked as not successful.

INFO: Building ZMQ
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near /home/panos/.julia/v0.6/ZMQ/deps/build.jl:36
INFO: Attempting to Create directory /home/panos/.julia/v0.6/ZMQ/deps/downloads
INFO: Downloading file https://github.com/zeromq/zeromq3-x/releases/download/v3.2.5/zeromq-3.2.5.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   608    0   608    0     0    608      0 --:--:-- --:--:-- --:--:--   708
100 2034k  100 2034k    0     0   508k      0  0:00:04  0:00:04 --:--:--  631k
INFO: Done downloading file https://github.com/zeromq/zeromq3-x/releases/download/v3.2.5/zeromq-3.2.5.tar.gz
INFO: Attempting to Create directory /home/panos/.julia/v0.6/ZMQ/deps/src
INFO: Attempting to Create directory /home/panos/.julia/v0.6/ZMQ/deps
INFO: Directory /home/panos/.julia/v0.6/ZMQ/deps already created
INFO: Attempting to Create directory /home/panos/.julia/v0.6/ZMQ/deps/builds/zmq
INFO: Changing Directory to /home/panos/.julia/v0.6/ZMQ/deps/builds/zmq
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
...
make[1]: Leaving directory '/home/panos/.julia/v0.6/ZMQ/deps/builds/zmq'
=======================================================================[ ERROR: ZMQ ]=======================================================================

LoadError: Provider BinDeps.BuildProcess failed to satisfy dependency zmq
while loading /home/panos/.julia/v0.6/ZMQ/deps/build.jl, in expression starting on line 36

============================================================================================================================================================

I think that somehow JuliaPro which uses custom folders to install packages is confusing BinDeps and/or ZMQ causing the installation to fail. How can this issue be resolved?


#2

try “sudo julia”?


#3

You can get some (possibly helpful) information about what’s going on by running:

using BinDeps
BinDeps.debug("ZMQ")

#4

On JuliaPro the result of BinDeps.debug(“ZMQ”) is:

julia> BinDeps.debug("ZMQ")
INFO: Reading build script...
The package declares 1 dependencies.
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near no file:0
 - Library "zmq"
    - Providers:
      - Autotools Build
      - BinDeps.AptGet package libzmq3 (can't provide)
      - BinDeps.Yum package czmq (can't provide)
      - BinDeps.Pacman package zeromq (can't provide)

Note that on Ubuntu there is no libzmq3 but libzmq5 installed in the system.
On the standard Julia version the result is different:

julia> BinDeps.debug("ZMQ")
INFO: Reading build script...
The package declares 1 dependencies.
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near no file:0
 - Library "zmq"
    - Satisfied by:
      - System Paths at /usr/lib/x86_64-linux-gnu/libzmq.so
    - Providers:
      - Autotools Build
      - BinDeps.AptGet package libzmq3 (can't provide)
      - BinDeps.Yum package czmq (can't provide)
      - BinDeps.Pacman package zeromq (can't provide)

Here it detects libzmq installed in the system and all goes well.
I noticed that ZMQ on JuliaPro comes with file ZMQ/deps/deps.jl which is:

# This is an auto-generated file; do not edit

# Pre-hooks

# Macro to load a library
macro checked_lib(libname, path)
    ((VERSION >= v"0.4.0-dev+3844" ? Base.Libdl.dlopen_e : Base.dlopen_e)(path) == C_NULL) && error("Unable to load \n\n$libname ($path)\n\nPlease re-run Pkg.build(package), and restart Julia.")
    quote const $(esc(libname)) = $path end
end

# Load dependencies
@checked_lib zmq "/lib64/libzmq.so.5"

# Load-hooks

On Ubuntu there is no /lib64 directory for libraries (this perhaps a Red Hat directory structure) and libzmq is installed in /usr/lib/x86_64-linux-gnu/. The same file in the standard Julia installation is different:

# This is an auto-generated file; do not edit

# Pre-hooks

# Macro to load a library
macro checked_lib(libname, path)
    ((VERSION >= v"0.4.0-dev+3844" ? Base.Libdl.dlopen_e : Base.dlopen_e)(path) == C_NULL) && error("Unable to load \n\n$libname ($path)\n\nPlease re-run Pkg.build(package), and restart Julia.")
    quote const $(esc(libname)) = $path end
end

# Load dependencies
@checked_lib zmq "/usr/lib/x86_64-linux-gnu/libzmq.so"

# Load-hooks

This one points to the right directory for libzmq.so. So, I replaced the file in JuliaPro with this one and repeated the debug command:

julia> BinDeps.debug("ZMQ")
INFO: Reading build script...
The package declares 1 dependencies.
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near no file:0
 - Library "zmq"
    - Providers:
      - Autotools Build
      - BinDeps.AptGet package libzmq3 (can't provide)
      - BinDeps.Yum package czmq (can't provide)
      - BinDeps.Pacman package zeromq (can't provide)

No progress. The next thing I did was to change file build.jl:

using BinDeps
using Compat

@BinDeps.setup

function validate(name, handle)
    try
        fhandle = Libdl.dlsym(handle, :zmq_version)
        major = Vector{Cint}(1)
        minor = Vector{Cint}(1)
        patch = Vector{Cint}(1)
        ccall(fhandle, Void, (Ptr{Cint}, Ptr{Cint}, Ptr{Cint}), major, minor, patch)
        return VersionNumber(major[1], minor[1], patch[1]) >= v"3"
    catch
        return false
    end
end

zmq = library_dependency("zmq", aliases = ["libzmq", "libzmq.so.3", "libzmq.so.4", "libzmq.so.5"], validate = validate)

provides(Sources, URI("https://github.com/zeromq/zeromq3-x/releases/download/v3.2.5/zeromq-3.2.5.tar.gz"), zmq)
provides(BuildProcess, Autotools(libtarget = "src/.libs/libzmq." * Libdl.dlext), zmq)

provides(AptGet, "libzmq3", zmq, os = :Linux)
provides(Yum, "czmq", zmq, os = :Linux)
provides(Pacman, "zeromq", zmq, os = :Linux)

if Compat.Sys.iswindows()
    using WinRPM
    provides(WinRPM.RPM, "zeromq", [zmq], os = :Windows)
elseif Compat.Sys.isapple()
    using Homebrew
    provides(Homebrew.HB, "zeromq@3.2", zmq, os = :Darwin)
end

@BinDeps.install Dict(:zmq => :zmq)

by replacting

provides(AptGet, "libzmq3", zmq, os = :Linux)

with

provides(AptGet, "libzmq5", zmq, os = :Linux)

Then I get the following:

julia> BinDeps.debug("ZMQ")
INFO: Reading build script...
The package declares 1 dependencies.
 - Library "zmq"
    - Providers:
      - Autotools Build
      - BinDeps.AptGet package libzmq5
      - BinDeps.Yum package czmq (can't provide)
      - BinDeps.Pacman package zeromq (can't provide)

So I start a Pkg.build(“ZMQ”) and I get:

julia> Pkg.build("ZMQ")
INFO: Building ZMQ
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near /home/panos/Applications/JuliaPro-0.6.1.1/JuliaPro/pkgs-0.6.1.1/v0.6/ZMQ/deps/build.jl:36
Installing dependency libzmq5 via `sudo apt-get install libzmq5`:
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libzmq5 is already the newest version (4.2.1-4ubuntu1).
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
================================================[ ERROR: ZMQ ]=================================================

LoadError: Provider BinDeps.PackageManager failed to satisfy dependency zmq
while loading /home/panos/Applications/JuliaPro-0.6.1.1/JuliaPro/pkgs-0.6.1.1/v0.6/ZMQ/deps/build.jl, in expression starting on line 36

===============================================================================================================

===============================================[ BUILD ERRORS ]================================================

WARNING: ZMQ had build errors.

 - packages with build errors remain installed in /home/panos/Applications/JuliaPro-0.6.1.1/JuliaPro/pkgs-0.6.1.1/v0.6
 - build the package(s) and all dependencies with `Pkg.build("ZMQ")`
 - build a single package by running its `deps/build.jl` script

===============================================================================================================

So, it tries to install libzmq5 using apt-get (already installed) and then it fails to load the library.
It looks like BinDeps cannot detect the library in folder /usr/lib/x86_64-linux-gnu when using JuliaPro.


#5

Anybody using JuliaPro on Ubuntu 16.04 or higher?


#6

I agree with your assessment. I wish I could help more, but I’ve never used Julia Pro and I’m not familiar with the way it searches for libraries.


#7

That is not a good idea. If you need this then your ~/.julia has the wrong permissions and that should be fixed instead.


#8

Issue in found only in MKL variant of JuliaPro. I was able to reproduce it and working on the fix


#9

Thank you.


#10

Issue is fixed and will be available in next version of JuliaPro.