BinDeps2 (Bbuilder+Bprovider)

Internals & Design
1

after reading some discussion on slack about using BinDeps2 (in the sense of: someone should move packages X and Y to BinDeps2) i’m a little puzzled by the whole story.

Is there anywhere a success story visible - as a library adaptation moved to use BinDeps2 for all 3 classic architetures?

2

See Sundials.jl

It uses BinaryBuilder.jl in its builder repo here:

and then the deps handling is done by BinaryProvider.jl. As an added bonus, it uses Clang.jl to auto-wrap the Sundials library as well, giving a 1-1 API with Sundials.

3

Snappy.jl and SnappyBuilder are the ones I got to work.

4

What are you puzzled by? Are you having trouble figuring out what the philosophy of BinDeps2 is or are you questioning whether it’s an effective approach?

5

I think it would really help if someone could do https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/131. For any slightly non-trivial case that is probably going to come up (it also came up on slack a couple of times), and it seems most people are a bit lost of how to pull that off (I know I am).

6

I think at one step of the wizard one can provide a build.jl for the dependency.

7

Yes, exacly, i have trouble figuring out what the philosophy of BinDeps2 is.

Installing binary dependencies by downloading from a pre-build database sounds similar like using classical package management. Just reinvented.
And putting the burden of creating the database (incl. cross-compilation) onto users sounds like a very optimistic target.

8

I can speak to this a little bit. First off, reading the BinaryBuilder documentation, (sparse as it is) a bit might fill in some cracks.

We are, indeed, trying to solve the same problem that package managers attempt to solve. However, there are many reasons why traditional package management is not a complete solution for us. Let’s start with what we want, the “statement of purpose” for all this technology: We want to be able to use third-party dependencies (C executables, Fortran libraries, etc…) with as little hassle on the user as possible.

Given that that’s what we want, let’s define some properties that an installation process should have for the user:

  • Should work on every platform Julia supports. (MacOS, Windows 32/64-bit, Linux with a bunch of different architectures, etc…)
  • Should not require sudo access
  • Should work well with Pkg3 ideals of being able to silo dependencies into an environment.
  • Should not require extensive compiler setups
  • Should be resilient against failure.
  • Should be uniform, e.g. package authors should not need to write code in their packages to make up for shortcomings of a binary dependency on a certain platform if we can help it.

Typical package managers tend to solve a subset of these problems very well, but as far as I’ve seen few of them solve all of them satisfactorily. As an example, Aptitude on Debian/Ubuntu does a lot of things right, but unfortunately doesn’t allow us to install things into arbitrary user directories, so there’s no way to have libfoo v0.1 installed for project Foo, and libfoo v0.2 installed for project Bar. Similarly, Nix is a nice platform that has a lot of good ideas, but we can’t use it on Windows or MacOS, and it doesn’t have binaries for everything, which would lead us into the direction of what we tried to do with Homebrew.jl, which I am desperately trying to move away from because it is too heavy of a maintenance burden to build these large systems on top of a base that can move out from underneath you.

In the end, we decided our usecase is specific and ambitious enough that it was worthwhile to reinvent the wheel a little bit so as to reduce complexity of binary object installation as much as possible. With the new system it is literally just downloading and unpacking a .tar.gz.

Some projects are an absolute nightmare to build. Most are pretty straightforward, once you’ve done it a few times. I invite you to look at my NettleBuilder repository’s build_tarballs.jl file to see if you still maintain that we are really putting a burden onto users. I rather think it is the other way around, where we are building the ability for users to create relocatable, generic binaries in a way that (to my knowledge) no other opensource programming language has ever attempted. Try building a c library for Python sometime; I spent three days trying to get a moderately sized c library to compile together as a .whl for Mac, Linux and Windows. It’s a nightmare. I have the utmost respect for those that are trying to improve the situation in the Python community right now, (I’ve even seen them look at what we’re trying to do) and I hope that this cross compilation approach can become something that other projects take note of as the easiest and most sane way to build binaries that can work anywhere.

To address David Anthoff, I unfortunately have stalled on my BinaryBuilder work due to real-life obligations so I don’t have an example of e.g. Ipopt using the result of OpenBLAS, but it will come.

10 Likes
9

I think it’s best to think of the design as following from the following axioms:

  1. Building on user machines is brittle so you have to provide binaries
  2. Using system package manager as we have done so far is inconvenient, because for every new binary dependency you want to add, you’d have to create package description files for at least 5 package registries, each with different versions of compiler, different formats for the registry, etc.

So, what BinaryBuilder provides is a tool chest for creating cross-compiled binary distributions for all major platforms and making sure you do that correctly (i.e. trying to bundle in and automate everything we’ve learned from building universally usable binaries for julia itself). That way you only have to write one recipe (for BinaryBuilder) rather than 5 and you can be sure that your users always have the correct version of the binary. I’d also like to claim that BinaryBuilder is by far the easiest way to create such a build recipe, but of course I’m biased. It also includes some extra sugar to set itself up as a travis repo and push binaries to GitHub releases, but you could host them anywhere. BinaryProvider is just a dumb download-this-tarball package.

Is this essentially creating our own Linux/Windows/OS X userspace distribution? Yes, but we’re ok with that if it means that installing binaries will just work, without any hassle no matter what system you’re using julia on.

11 Likes