Overriding BinaryProvider download to use local/custom build?

Is there a standard way of overriding BinaryProviders download mechanism and make it use a custom build? I’ve encountered two cases where that would be very valuable:

  • A custom build is necessary (e.g. I may need to build HDF5 with some specific options for my application).

  • Not all platform are supported by the standard BinaryBuilder configs, e.g. Julia on Android: Runs fine via its-pointless repository for termux, but it’s very limited since basic packages like Distributions can’t be installed without binary deps).

Assuming I do have a local installation of the binary dependency, in a compatible version, is there any way to tell BinaryProvider to use it? In past, with packages using BinDeps, there was usually a way to do that.

2 Likes

I don’t think there’s a generic way to do it for any given repo, but assuming you can change the build script, you can have it do anything you want at the build stage. Eg here, SCS allows the user to set an environmental variable to override binary builder and use a custom binary: https://github.com/JuliaOpt/SCS.jl/blob/9b4959717636b8fdf9179a91490fdabfe03690ad/deps/build.jl#L64

The Pkg.jl package that will be shipped with julia v1.3 will allow you to override artifacts: https://julialang.github.io/Pkg.jl/dev/artifacts/#Overriding-artifact-locations-1. For the time being you can look at what SpecialFunctions.jl does: https://github.com/JuliaMath/SpecialFunctions.jl/blob/v0.8.0/deps/build.jl

1 Like

The result of building a package Foo is usually a deps/deps.jl file that defines a const libfoo = "...path to libfoo.so....". So one relatively easy way to point a package at a custom binary is edit/overwrite deps.jl to change the value of this constant. Just do using Foo; pathof(Foo) to find out where the module Foo is installed, and then edit its deps.jl file to use the path of the library that you want (or create a deps.jl file if the build failed.)

(If you are creating a new deps.jl file because the build failed, note that deps.jl also contains a function check_deps() that checks that the library can be opened. In a pinch you can just define check_deps() = nothing.)

1 Like

The Pkg.jl package that will be shipped with julia v1.3 will allow you to override artifacts

Oh, nice! It’ll take a while before the new artifacts feature will be in wide use though, I guess. From what I understand, this is intended to replace BinaryProvider eventually, correct?

The result of building a package Foo is usually a deps/deps.jl

Oh, sure, hacking “deps.jl” is always an option, and I’ve used it for demos. But it’s of course not something I can recommend to a group of people that may need to use our code with specific HDF5 options. :wink:

Yes, that’s correct. There will be no deps/build.jl at all, you just depend on the binary library in Project.toml. See this WIP blog post: https://github.com/JuliaLang/www.julialang.org/pull/417/files or how this works in practice in Cairo.jl: https://github.com/JuliaGraphics/Cairo.jl/pull/293

3 Likes

Oh, this is very nice - can’t wait for this to be widely adopted!