Build Julia on NixOS

Hello!

Here I would like to create a topic about the use of Julia on NixOS, and open the contact with other Julia users under NixOS or those using the Nix package manager. First, just a short introduction about why the use of Julia with this operating system might not be so trivial:

NixOS is a distribution that has a different way to install packages to have a kind of purely reproducible working environment. This means that it doesn’t follow the FHS (file hierarchical system) of unix/linux.

To give an example of what it implies in the case of python, a typical installation with pip or conda of a python package might not work, since it is usually looking for files on specific directories (on a prescribed FHS like /usr/lib, /usr/share/ … or similar) to resolve dependencies. Actually, it happens that these dependencies are (or can be) in the system but not there (when conda/pip are looking for them).

Julia compiler itself can be easily installed (as simple as: nix-env -iA nixos.julia_11 for version 1.1) but the Julia’s built-in package manager usually has difficulties to build the packages and organize their dependencies. The NixOS people provided a solution for those cases, as in this operating system it is possible to run a program on a virtual environment that fulfil the FHS. This is useful to use, for example, conda on NixOS on a smooth way, or installations depending on it (for example here.

More and less this is the idea or direction to overcome the problem also in Julia, but still the users may need some support, so I have open this topic to share our experiences or difficulties. Here I give some links to discussion about Julia within the NixOS community:

Inside Julia world we had discussions about support of Nix package manager:

The current solution, based on create an FHD environment, that aims to solve our problems is close to this piece of code to be added in our NixOS configuration of overlays (kindly provided by other user):

{pkgs, stdenv, ...}:

with pkgs;
let
  julia = julia_11;
  d = version: "v${lib.concatStringsSep "." (lib.take 2 (lib.splitString "." version))}";
  extraLibs = [
    # ImageMagick.jl ==========================
    imagemagickBig
    # HDF5.jl =================================
    hdf5
    # Cairo.jl ================================
    cairo gettext pango.out glib.out
    # Gtk.jl ==================================
    gtk3 gdk_pixbuf
    # GZip.jl required by DataFrames.jl =======
    gzip zlib
    # GR.jl which runs without Xrender & Xext =
    # but cannot save files ===================
    xorg.libXt xorg.libX11 xorg.libXrender xorg.libXext glfw freetype
    # Flux.jl =================================
    cudatoolkit linuxPackages.nvidia_x11 git gitRepo gnupg autoconf curl
    procps gnumake utillinux m4 gperf unzip libGLU_combined ncurses5 stdenv.cc binutils
    xorg.libXi xorg.libXmu freeglut xorg.libXext xorg.libX11 xorg.libXv xorg.libXrandr zlib
    # Arpack.jl ===============================
    arpack gfortran.cc
    (pkgs.runCommand "openblas64_" {} ''
      mkdir -p "$out"/lib/
      ln -s ${openblasCompat}/lib/libopenblas.so "$out"/lib/libopenblas64_.so.0
    '')
  ];
in
stdenv.mkDerivation rec {
  name = "julia-env";
  version = julia.version;
  nativeBuildInputs = [ makeWrapper cacert git pkgconfig which ];
  buildInputs = [
    julia
    /* jupyterEnv  # my custom jupyter */
  ] ++ extraLibs;
  phases = [ "installPhase" ];
  installPhase = ''
    export CUDA_PATH="${cudatoolkit}"
    export LD_LIBRARY_PATH=${lib.makeLibraryPath extraLibs}
    # pushd $JULIA_PKGDIR/${d version}
    makeWrapper ${julia}/bin/julia $out/bin/julia \
        --prefix LD_LIBRARY_PATH : "$LD_LIBRARY_PATH" \
        --prefix LD_LIBRARY_PATH ":" "${linuxPackages.nvidia_x11}/lib" \
        --set CUDA_PATH "${cudatoolkit}" \
        --set JULIA_PKGDIR $JULIA_PKGDIR
  '';
}

Kind regards!

20 Likes

This is a great summary of the state of the current discussion. Thanks.

A couple of minor things, the expression I posted in the other thread and you copied here, is not setting up a user FHS. For that you would need to use buildFHSUserEnv instead of mkDerivation.

Also I am not the original author of that expression. I have seen variations of it posted in various threads so I can’t really remember where I exactly got it from but credit should probably go to tbenst.

Thanks @cstich, you are right it is a derivation rather than a FHS user environment. Here tbenst, gave the FHS user environment approach.

In Nix/NixOS usually the packages are built as a derivation, which a basic block in which the nix package manager, and also it is possible to override or overlays a given derivation to personalize/improve the build of package.

I haven’t read this in detail yet but it’s great you are doing it - every time I think about using Julia I then recall I can’t use nix and then give up. Thank you once again.

https://github.com/NixOS/nixpkgs/pull/73261 so this script no longer works

How is this supposed to work? AFAICS $JULIA_PKGDIR is the empty string so when I start the wrapper, none of the packages are available.

Hello @idontgetoutmuch,

Maybe you can open also a question at nixos-discurse and refer to this topic here. I did not try the code above since I moved to NixOS 20.03, but I remember that I had sometimes problems when Julia Pkg tried to install and manage external decencies (python, etc).

In nixos-discurse you can certainly contact with the person who did the piece of code above, and they are very helpful.

Cheers.

1 Like

Can I just check that you run Julia via nix but install the Julia packages using the Julia package manager? Only the external dependencies e.g. lapack are managed by nix?

This seems very un-nix to me. For example with R I can manage the R packages like this

with pkgs;
let
  R-with-my-packages = rWrapper.override{ packages = with rPackages; [ ggplot2 dplyr xts ]; };
in ...

I can do the same with Python and Haskell but there seems no equivalent way to do this with Julia.

Yes. As far as I know: not only me but all of us. We install Julia with nix (or nixos) but additional Julia packages are installed using Pkg of Julia.

I understand that to be able to use the nix way, we would need people that create (and maintain) derivations for Julia packages/modules with all the dependencies included. This means a lot of work and human resources that at this moment I think we don’t have. Maybe you know better than me if in a future we will have something similar to pip2nix to create derivations for Julia that can help us in the process.

On the other side, the Pkg of Julia itself is nice. So, currently there are two approaches: create a FHS environment (something similar to what is done for conda in nixos) or an override/overlays.

@idontgetoutmuch, if you contact at nixos-discourse with the author of the wrapper above, then you could eventually, referred here to a possible updated information.

Cheers and thanks!

Thanks - although I managed to get Julia to use the MPI I wanted, it seems to have disregarded my attempts to make it use the SUNDIALS I wanted.

I can see why Juia packages up 3rd party packages the way it does but I think it’s a mistake not to have an interface that lets nix handle the package management for you.

For me not being able to use nix is a big downside.

There might be scope to build something like nix-autobahn to take the pain out of managing the external dependencies for a julia project with nix or alternatively build something like the FHS steam-run but for julia that just works.

Hi @cstich,

I did not know about nix-autobahn but it seems a nice idea/tool. My view regarding Julia Packages management with nix/NixOS is that it involves two aspects:

  1. Manage external dependencies in such way that Julia Pkg manager is able to find and use correctly those external dependencies.
  2. How to install/update/remove the Julia packages themselves: using Julia Pkg or implement directly a nix approach.

So (1) is the focus right now, and it is more and less resolved although still with space to improve, like this nix-autobahn or similar approaches.

The (2), as I understood, was commented by @idontgetoutmuch like manage directly the Julia Packages with nix. I don’t know if there is any plan for it (or even if it was considered), but I understood from Julia Pkgs and NixOS that right now the focus in (1).

By the way, @cstich, explained GR+Julia working in NixOS, how to patch the gksqt binary for Julia in NixOS if anyone need it.

Cheers!

@RCHG

Yes, (1) definitely seems to be the current approach for using nix and julia. Although it is a bit of faff to go hunting for external dependencies. At least most packages seem to be okay with nix provided dependencies.

AFAIK with respect to (2) there is just really not the manpower within nix to repackage even parts of the julia ecosystem at the moment. The julia nix expression has needed a re-write for quite some time now and that has not happened. Even the qksqt patch is a bit of a hack. Someone actually should package that. At least nix now has a somewhat current julia version.

2 Likes

I wasn’t able by now to use either of the various julia / julia-env derivations with nix though I would love to.

It was always failing tests, so my question would be is there any chance to disable tests and if so can you point me to an example of how this is done ?

I did try setting checkPhase = "true"; but that didn’t do much difference.

I gave up in the end. I wanted to use a particular version of SUNDIALS but Julia / the Julia package manager didn’t want to let me. Currently I back with Haskell.

Next thing I tried was to disable tests via overlays.

# ~/.config/nixpkgs/overlays/julia.nix
self: super: {
  julia_11 = super.julia_11.overrideDerivation(oldAttrs: {
      doCheck = false;
    });
  arpack = super.arpack.overrideDerivation(oldAttrs: {
      doCheck = false;
    });
}

with this expression (usage: nix-build, see comment here):

# ~/.config/nixpkgs/program/julia/default.nix
with import <nixpkgs> { };                                                                                                                                                                                                                      let
  extraLibs = [                                                                                                             arpack
    gfortran.cc
    (pkgs.runCommand "openblas64_" { } ''
      mkdir -p "$out"/lib/
      ln -s ${openblasCompat}/lib/libopenblas.so "$out"/lib/libopenblas64_.so.0
    '')
  ];
in stdenv.mkDerivation rec {
  name = "julia-env";
  version = julia.version;
  nativeBuildInputs = [ makeWrapper cacert git pkgconfig which ];
  buildInputs = [ julia_11 ] ++ extraLibs;
  phases = "installPhase";
  checkPhase = "true";
  installCheckPhase = "false";
  installPhase = ''
    export LD_LIBRARY_PATH=${lib.makeLibraryPath extraLibs}
    makeWrapper ${julia}/bin/julia $out/bin/julia \
        --prefix LD_LIBRARY_PATH : "$LD_LIBRARY_PATH" \
        --set JULIA_PKGDIR $JULIA_PKGDIR
  '';
}

I also tried based on this to deactivate the tests via overlay.

# ~/.config/nixpkgs/overlays/julia.nix 
self: super: {
  julia_11 = super.julia_11.overrideDerivation(oldAttrs: {
      doCheck = false;
    });
  arpack = super.arpack.overrideDerivation(oldAttrs: {
      doCheck = false;
    });
}

with this (usage: nix-build etc.):

# ~/.config/nixpkgs/program/julia/default.nix
with import <nixpkgs> { };                                                                                                                                                                                                                      let
  extraLibs = [
    arpack
    gfortran.cc
    (pkgs.runCommand "openblas64_" { } ''
      mkdir -p "$out"/lib/
      ln -s ${openblasCompat}/lib/libopenblas.so "$out"/lib/libopenblas64_.so.0
    '')
  ];
in stdenv.mkDerivation rec {
  name = "julia-env";
  version = julia.version;
  nativeBuildInputs = [ makeWrapper cacert git pkgconfig which ];
  buildInputs = [ julia_11 ] ++ extraLibs;
  phases = "installPhase";
  checkPhase = "true";
  installCheckPhase = "false";
  installPhase = ''
    export LD_LIBRARY_PATH=${lib.makeLibraryPath extraLibs}
    makeWrapper ${julia}/bin/julia $out/bin/julia \
        --prefix LD_LIBRARY_PATH : "$LD_LIBRARY_PATH" \
        --set JULIA_PKGDIR $JULIA_PKGDIR
  '';
}

Can you not just set checkTarget = false in your overlay?

@cstich checkTarget = false does not prevent the tests from running as it seems.

Currently building (nix-build, tests are running) with buildInputs = [ julia_11 ] ++ extraLibs; etcetera in default.nix (extraLibs = [];) as well as this overlay:

self: super: {
  julia_11 = super.julia_11.overrideAttrs (oldAttrs: {
    doCheck = false;
    doInstallCheck = false;
    checkTarget = false;
    });
}

I do plan to try this suggestion though I’m not shure if I will find time for it this week.

I am not sure how helpful it is, but I am using this expression:

{ stdenv, fetchurl, fetchzip, fetchFromGitHub
# build tools
, gfortran, m4, makeWrapper, patchelf, perl, which, python2
, cmake
# libjulia dependencies
, libunwind, readline, utf8proc, zlib
# standard library dependencies
, curl, fftwSinglePrec, fftw, gmp, libgit2, 
mpfr, openlibm, openspecfun, pcre2
# linear algebra
, openblas, arpack
# llvm
, llvm
}:

with stdenv.lib;

# All dependencies must use the same OpenBLAS.
let
  arpack_ = arpack;
in
let
  arpack = arpack_.override { inherit openblas; };
in

let 
  majorVersion = "1";
  minorVersion = "5";
  maintenanceVersion = "0";
  src_sha256 = "1x10q46q2bkr19nqgc05kx8sachh8dz8v7qrvqwdagr4g8yxmqrs";
  version = "${majorVersion}.${minorVersion}.${maintenanceVersion}";
in 

stdenv.mkDerivation rec {
  pname = "julia";
  inherit version;

  src = fetchzip {
     url = "https://github.com/JuliaLang/julia/releases/download/v1.5.0-rc1/julia-1.5.0-rc1-full.tar.gz";
     sha256 = src_sha256;
   };

  prePatch = ''
    export PATH=$PATH:${cmake}/bin
    '';

  patches = [
    ./use-system-utf8proc-julia-1.3.patch

    # Julia recompiles a precompiled file if the mtime stored *in* the
    # .ji file differs from the mtime of the .ji file.  This
    # doesn't work in Nix because Nix changes the mtime of files in
    # the Nix store to 1. So patch Julia to accept mtimes of 1.
    ./allow_nix_mtime.patch
  ];

  postPatch = ''
    patchShebangs . contrib
    for i in backtrace cmdlineargs; do
      mv test/$i.jl{,.off}
      touch test/$i.jl
    done
    rm stdlib/Sockets/test/runtests.jl && touch stdlib/Sockets/test/runtests.jl
    rm stdlib/Distributed/test/runtests.jl && touch stdlib/Distributed/test/runtests.jl
    # LibGit2 fails with a weird error, so we skip it as well now
    rm stdlib/LibGit2/test/runtests.jl && touch stdlib/LibGit2/test/runtests.jl
    sed -e 's/Invalid Content-Type:/invalid Content-Type:/g' -i ./stdlib/LibGit2/test/libgit2.jl
    sed -e 's/Failed to resolve /failed to resolve /g' -i ./stdlib/LibGit2/test/libgit2.jl
  '';

  buildInputs = [
    arpack fftw fftwSinglePrec gmp libgit2 libunwind mpfr
    pcre2.dev openblas openlibm openspecfun readline utf8proc
    zlib
  ]
  ++ stdenv.lib.optionals stdenv.isDarwin [CoreServices ApplicationServices]
  ;

  nativeBuildInputs = [ curl gfortran m4 makeWrapper patchelf perl python2 which ];

  makeFlags =
    let
      arch = head (splitString "-" stdenv.system);
      march = { x86_64 = stdenv.hostPlatform.platform.gcc.arch or "x86-64"; i686 = "pentium4"; }.${arch}
              or (throw "unsupported architecture: ${arch}");
      # Julia requires Pentium 4 (SSE2) or better
      cpuTarget = { x86_64 = "x86-64"; i686 = "pentium4"; }.${arch}
                  or (throw "unsupported architecture: ${arch}");
    in [
      "ARCH=${arch}"
      "MARCH=${march}"
      "JULIA_CPU_TARGET=${cpuTarget}"
      "PREFIX=$(out)"
      "prefix=$(out)"
      "SHELL=${stdenv.shell}"

      "USE_SYSTEM_BLAS=1"
      "USE_BLAS64=${if openblas.blas64 then "1" else "0"}"
      "LIBBLAS=-lopenblas"
      "LIBBLASNAME=libopenblas"

      "USE_SYSTEM_LAPACK=1"
      "LIBLAPACK=-lopenblas"
      "LIBLAPACKNAME=libopenblas"

      "PCRE_CONFIG=${pcre2.dev}/bin/pcre2-config"
      "PCRE_INCL_PATH=${pcre2.dev}/include/pcre2.h"
      "USE_SYSTEM_READLINE=1"
      "USE_SYSTEM_UTF8PROC=1"
      "USE_SYSTEM_ZLIB=1"

      "USE_BINARYBUILDER=0"
    ];

  LD_LIBRARY_PATH = makeLibraryPath [
    arpack fftw fftwSinglePrec gmp libgit2 mpfr openblas openlibm
    openspecfun pcre2
  ];

  enableParallelBuilding = true;

  doCheck = !stdenv.isDarwin;
  checkTarget = false;
  # Julia's tests require read/write access to $HOME
  preCheck = ''
    export HOME="$NIX_BUILD_TOP"
  '';

  preBuild = ''
    sed -e '/^install:/s@[^ ]*/doc/[^ ]*@@' -i Makefile
    sed -e '/[$](DESTDIR)[$](docdir)/d' -i Makefile
    export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
  '';

  postInstall = ''
    # Symlink shared libraries from LD_LIBRARY_PATH into lib/julia,
    # as using a wrapper with LD_LIBRARY_PATH causes segmentation
    # faults when program returns an error:
    #   $ julia -e 'throw(Error())'
    find $(echo $LD_LIBRARY_PATH | sed 's|:| |g') -maxdepth 1 -name '*.${if stdenv.isDarwin then "dylib" else "so"}*' | while read lib; do
      if [[ ! -e $out/lib/julia/$(basename $lib) ]]; then
        ln -sv $lib $out/lib/julia/$(basename $lib)
      fi
    done
  '';

  passthru = {
    inherit majorVersion minorVersion maintenanceVersion;
    site = "share/julia/site/v${majorVersion}.${minorVersion}";
  };

  meta = {
    description = "High-level performance-oriented dynamical language for technical computing";
    homepage = https://julialang.org/;
    license = stdenv.lib.licenses.mit;
    maintainers = with stdenv.lib.maintainers; [ raskin rob garrison ];
    platforms = [ "i686-linux" "x86_64-linux" "x86_64-darwin" ];
    broken = stdenv.isi686;
  };
}

to build julia 1.5 locally. You can get the patches from nixpkgs.

4 Likes