Build Julia on NixOS

The buildFHSUserEnv that was working for me on 1.5.3 also works on 1.6-beta1 as far as I can tell. I just built some projects of mine with tons of dependencies (including DataFrames, MLJ and Gadfly) and had no issues whatsoever. Since it uses buildFHSUserEnv, it is a bit of a hack, but at least it works.

I put the derivation and a README at https://github.com/rikhuijzer/julia-nixos.

2 Likes

Does anyone have julia2nix working on macos (big sur)?

1 Like

This is great news :grinning: but Iā€™d still like what I have in R or Python e.g. the ability to list the packages in the nix derivation itself

{ pkgs ? import <nixpkgs> { config.allowBroken = true; } }:

let

  julia = pkgs.julia_15.override {
    # stdenv = pkgs.overrideCC pkgs.stdenv pkgs.gcc;
    # blas = pkgs.openblas;
  };

in

pkgs.mkShell {
  buildInputs = [
    julia
    pkgs.python3
    pkgs.python3Packages.numpy

  ];

  MYVARIABLE = "hello";
}

Is that ever going to be possible?

In case anyone is interested, Iā€™ve published the code I use to package Julia with a precompiled sysimage.

You can try it out via my Nix flake. Note that it takes a long time to initially compile:

nix shell github:aaronjanse/dotfiles#julia

https://github.com/aaronjanse/dotfiles/tree/master/pkgs/julia

1 Like

FYI I now have it working with lorri / direnv / emacs so that when I enter the directory I get automatically switched to the desired environment.

Sorry for the delay, missed this question. Yes, I think itā€™s possible. The holy grail would be to check in a comprehensive package set to Nixpkgs (consisting of all the packages in the general registry).

It would be some work to set up the tools to build this set and update it in Nixpkgs on a recurring basis the way Python etc. do.

One difficulty I could foresee is that Julia is unusually precise about its package dependencies. When you ask Julia to install a set of packages, I think it does some complex constraint solving to find a set of versions that makes every dependency happy. So any Nix scheme would probably have to contain similar logicā€“you wouldnā€™t be able to just have a single version of each package the way some language ecosystems do.

But on the whole I think the task is straightforward, it just needs someone to put some time into it.

1 Like

What about trying to get your julia FHS merged into nixpkgs? Currently the user experience with julia and Pkg3 is a bit abysmal on nix. People have taken a similar approach getting conda to work with nix. I think this would be good to have at least until we figure out a better way to support and build julia and the julia package ecosystem.

I also donā€™t mind issuing a pull request myself but I wanted to check with you first.

1 Like

I thought that there was a good reason for not using this approach, so that is why I didnā€™t try to get it into nixpkgs. I would appreciate it very much if you could look into it and you donā€™t have to give me credits. :grin: I also copied parts of it from someone but unfortunately donā€™t know who. Also, I think he copied it from somewhere and Iā€™ve made some changes.

Sure, I can take it on. Ideally, yes, there would be more nix way of doing things and julia2nix definitely is a big step forward with respect to usability, but I still think itā€™s worth having something that just works.

FWIW, I also got 1.6 to build now from source, although that expression is in dire need of refactoring and fails a good chunk of itā€™s tests.

3 Likes

I believe @ninjin is looking into this :eyes:

2 Likes

Thanks for flagging this to me @giordano!

Indeed, I have cleaned-up expressions for 1.0.x and 1.5.x in the pipeline that I hope to have cleared and in PRs over the weekend. I really should make my Nixpkgs fork and Julia with Nix fork public and post them here as well in case you guys want to provide feedback and/or patches.

@cstich: If you are up for it, would you be happy to share that working 1.6rc expression? I have not yet taken a stab at the 1.6rc just yet, but I would greatly appreciate you sharing any quirks you had to deal with moving from 1.5.4. What I have done so far is to have a quick look at the 1.6rc code for Fedora a few days ago.

4 Likes

Thatā€™s great news. Let me know if I can help in any way. I am happy to have a look at the (eventual) pull request as well.

Sure, no problem. Here is my working expression for 1.6:

1 Like

Late to the party, but for those interested @ninjin created the #nix channel on the Julia Slack for those interested. I put together a Google Doc summarizing some of the conversations around Julia + Nix that have been happening and linked it there.

1 Like

Any chance we could move to the Julia Zulip instead? Zulip solves the problems of named topics, searchability, and losing history into the slackhole, while retaining the chatty feel.

@jzr Zulip looks interesting (especially the unlimited history), but I think most of the core Julia devs (specifically those familiar with Pkg/Artifacts/BinaryBuilder) are primarily on Slack. Those folks have been super helpful thus far.

After a tremendous amounts of efforts (note: non of which by me), Julia 1.6 is now in nixpkgs (julia: init julia_10-bin and julia_16-bin by ninjin Ā· Pull Request #123188 Ā· NixOS/nixpkgs Ā· GitHub). This will end up in NixOS 21.05. Iā€™ve been running Pontusā€™ binary for more than a week now and it works great :partying_face:

Many many thanks to @ninjin and the other contributors

8 Likes

My pleasure, although it still irks me that the v1.6.x source builds are not in a working state. Perhaps I can resume hacking on this in a few weeks if I can steal the time.

2 Likes

Out of curiosity: what is exactly the benefit of having a more nix way to manage packages in Julia? Unlike Python, Ruby and R, for example, Julia has a reliable and built-in package manager which can handle multiple versions of the same package. (The depot can even handle multiple versions of Julia.)

1 Like

I am not an expert but I think that it is not so simple to use the Julia package manager in NixOS (not like in other Linux distributions). Pkg3 of Julia sometimes is looking for external libraries, or python code etcā€¦ And, at least in the previous versions of Julia in NixOS that I tested, this can be an issue. Often these libraries are not found by Pkg3 even when they are installed in the system because NixOS has a different location of the packages. For example, if Pkg3 is trying to install/use matplotlib for Plots using conda this might not work for scratch.

There are several approaches to overcome this problem like julia FHS or during the compilation of Julia itself, but a julia2nix would be something nice as it could introduce the full tree of dependences of each package to ensure that everything is working smoothly in NixOS.

1 Like

@rikh There are issues with some individual binaries, as @RCHG alludes to. Thereā€™s also the more general point that once youā€™ve found Nix nirvana, you donā€™t want to go back to every little thing having its own package manager.

1 Like