I can speak to this a little bit. First off, reading the BinaryBuilder documentation, (sparse as it is) a bit might fill in some cracks.
We are, indeed, trying to solve the same problem that package managers attempt to solve. However, there are many reasons why traditional package management is not a complete solution for us. Let’s start with what we want, the “statement of purpose” for all this technology: We want to be able to use third-party dependencies (C executables, Fortran libraries, etc…) with as little hassle on the user as possible.
Given that that’s what we want, let’s define some properties that an installation process should have for the user:
- Should work on every platform Julia supports. (MacOS, Windows 32/64-bit, Linux with a bunch of different architectures, etc…)
- Should not require
- Should work well with Pkg3 ideals of being able to silo dependencies into an environment.
- Should not require extensive compiler setups
- Should be resilient against failure.
- Should be uniform, e.g. package authors should not need to write code in their packages to make up for shortcomings of a binary dependency on a certain platform if we can help it.
Typical package managers tend to solve a subset of these problems very well, but as far as I’ve seen few of them solve all of them satisfactorily. As an example, Aptitude on Debian/Ubuntu does a lot of things right, but unfortunately doesn’t allow us to install things into arbitrary user directories, so there’s no way to have
libfoo v0.1 installed for project Foo, and
libfoo v0.2 installed for project Bar. Similarly,
Nix is a nice platform that has a lot of good ideas, but we can’t use it on Windows or MacOS, and it doesn’t have binaries for everything, which would lead us into the direction of what we tried to do with Homebrew.jl, which I am desperately trying to move away from because it is too heavy of a maintenance burden to build these large systems on top of a base that can move out from underneath you.
In the end, we decided our usecase is specific and ambitious enough that it was worthwhile to reinvent the wheel a little bit so as to reduce complexity of binary object installation as much as possible. With the new system it is literally just downloading and unpacking a
Some projects are an absolute nightmare to build. Most are pretty straightforward, once you’ve done it a few times. I invite you to look at my NettleBuilder repository’s build_tarballs.jl file to see if you still maintain that we are really putting a burden onto users. I rather think it is the other way around, where we are building the ability for users to create relocatable, generic binaries in a way that (to my knowledge) no other opensource programming language has ever attempted. Try building a c library for Python sometime; I spent three days trying to get a moderately sized c library to compile together as a .whl for Mac, Linux and Windows. It’s a nightmare. I have the utmost respect for those that are trying to improve the situation in the Python community right now, (I’ve even seen them look at what we’re trying to do) and I hope that this cross compilation approach can become something that other projects take note of as the easiest and most sane way to build binaries that can work anywhere.
To address David Anthoff, I unfortunately have stalled on my BinaryBuilder work due to real-life obligations so I don’t have an example of e.g. Ipopt using the result of OpenBLAS, but it will come.