Can Pkg.build on the canonical terminal REPL and inside VSCode REPL, but not with Atom/Juno's REPL

compilation

#1

I’m unable to use or build certain packages when using Atom’s REPL (Juno), but can use/build the packages when launching Julia from my terminal or when using the REPL from inside VSCode. For example,
issuing the following commands from inside the REPL of the Atom/Juno editor:

              _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.2 (2017-12-13 18:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> Pkg.build("HDF5")
INFO: Building Blosc
INFO: Building HDF5
==========================================[ ERROR: HDF5 ]===========================================

LoadError: could not spawn `apt-cache showpkg hdf5-tools`: no such file or directory(ENOENT)
while loading /home/zygmunt/.julia/v0.6/HDF5/deps/build.jl, in expression starting on line 40

====================================================================================================

==========================================[ BUILD ERRORS ]==========================================

WARNING: HDF5 had build errors.

 - packages with build errors remain installed in /home/zygmunt/.julia/v0.6
 - build the package(s) and all dependencies with `Pkg.build("HDF5")`
 - build a single package by running its `deps/build.jl` script

====================================================================================================

whereas issuing the same command from the Julia REPL launched directly from my Linux terminal gives:

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.2 (2017-12-13 18:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> Pkg.build("HDF5")
INFO: Building Blosc
INFO: Building HDF5

julia> 

julia> Pkg.test("HDF5")
INFO: Testing HDF5
Test Summary: | Pass  Total
plain         |  126    126
Test Summary: | Pass  Total
readremote    |   23     23
Test Summary: | Pass  Total
extendtest    |   15     15
Test Summary: | Pass  Total
gc            |  100    100
Test Summary: | Pass  Total
external      |    6      6
INFO: HDF5 tests passed

After building the package from the canonical REPL I still cannot use it inside Atom/Juno. I get the following
error:

Starting Julia...
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.2 (2017-12-13 18:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> Pkg.test("HDF5")
INFO: Testing HDF5
ERROR: LoadError: InitError: error compiling __init__: could not load library "/usr/lib/x86_64-linux-gnu/libhdf5_serial.so.10"
/usr/lib/x86_64-linux-gnu/libhdf5_serial.so.10: cannot open shared object file: No such file or directory
Stacktrace:
 [1] _include_from_serialized(::String) at ./loading.jl:157
 [2] _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:200
 [3] _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:236
 [4] _require(::Symbol) at ./loading.jl:441
 [5] require(::Symbol) at ./loading.jl:405
 [6] include_from_node1(::String) at ./loading.jl:576
 [7] include(::String) at ./sysimg.jl:14
 [8] process_options(::Base.JLOptions) at ./client.jl:305
 [9] _start() at ./client.jl:371
during initialization of module HDF5
while loading /home/zygmunt/.julia/v0.6/HDF5/test/runtests.jl, in expression starting on line 1
==========================================[ ERROR: HDF5 ]===========================================

failed process: Process(`/media/zygmunt/big/Julia/bin/julia -Cx86-64 -J/media/zygmunt/big/Julia/lib/julia/sys.so --compile=yes --depwarn=yes --check-bounds=yes --code-coverage=none --color=yes --compilecache=yes /home/zygmunt/.julia/v0.6/HDF5/test/runtests.jl`, ProcessExited(1)) [1]

====================================================================================================
ERROR: HDF5 had test errors

There is a known issue with Atom 1.26.0 (and other versions) in that it does not load the PATH properly. I’ve been able to fix issues I’ve had with the usage of other packages by adding my path variables explicitly in the Edit--->Init Script file:

process.env.PATH = ['/home/zygmunt/bin:/home/zygmunt/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:', process.env.PATH].join(':')

However, this seems to be insufficient for resolving the HDF5 problem. Is there some other environment variable that I should be explicitly including?

I have tried push!(Libdl.DL_LOAD_PATH, "/usr/lib/x86_64-linux-gnu") as per the suggestion of the HDF5 installation instructions but this had no effect.

Following on from the installation instructions, I also cannot see the environment variable LD_LIBRARY_PATH defined when I do ; echo $ENV from either the canonical or Juno REPL. Hence, I am not sure what precisely Juno is blocking that is preventing the usage and compilation of this package. Many thanks for any suggestions you may have.


#2

I’ve figured out what the problem is, and I suspect that other novice users may hit this problem in the future as well. I am using Linux Mint, and I had installed the Atom editor via the built-in application manager. The application manager for Linux Mint uses something called Flatpak to launch Atom. Flatpak creates a sandboxed environment in which Atom runs. It prevents the Atom session from seeing or accessing many critical parts of the system, such as /usr/local/bin or /usr/bin. It also clobbers the PATH variable. Hence, if I shell out in the Julia REPL inside Atom, I am unable to execute apt or apt-cache because it doesn’t exist in the sandbox.

According to the documentation of Flatpak, there is a way to launch Atom from Flatpak so that it has access to the host file-system, with the exception of
/lib, /lib32, /lib64, /bin, /sbin, /usr, /boot, /root, /tmp, /etc, /app, /run, /proc, /sys, /dev, /var

which, ironically, are the directories that we need.

@nalimilan I noticed that you had looked a bit into Flatpak and Atom in this issue. Do you think it will be possible to circumvent the lack of access to apt and apt-cache somehow?


#3

I think the way to go is to port packages to BinaryProvider (instead of the current BinDeps system), so that needed libraries are simply downloaded and installed inside the Flatpak sandbox. That’s the plan anyway, so it’s good it fits the Flatpak model.

BTW, how is the sandboxed Atom able to start the julia executable? Is it installed inside the sandbox too? That’s tricky to handle with Flatpak, and so far my question to the mailing list hasn’t got any reply.


#4

I think I managed to unwittingly sidestep the julia executable issue because I happened not to install julia in the any of the blacklisted directories. I happened to install it on a separate disk, which for some reason is being picked up as a removable media. I had configured Juno in Atom to point to Julia on this removable media and it just worked. I didn’t even know at the time that Atom was running inside a sandbox.