Binary dependencies

I was pointing at conda-forge (which is ~2yo) as an example that language-agnostic approach is more likely to scale, and would help making Julia more inter-operable with other large code bases.

I am advocating

  • for the official download bundle Julia to ship a simple installation prefix including Julia as a package installed among others.
  • that this installation prefix may be used to install other assets.
  • that whatever underlies Pkg.add installs non-julia dependencies as normal packages in that installation prefix.

I think we’re talking past each other here. What I’m addressing is how to support binary dependencies installed by the system package manager (or an HPC administrator) and re-using them from within julia.

This is how things work with BinaryBuilder (the actual binary packages are just .tar.gz files with standard prefix layout), with the additional complication that the prefix is sharded to allow for quick switching between versions (of course you could implement this the other way, with having a single prefix that gets put together every time you activate an environment, but that’s an implementation detail)

Ok, this sounds like the conda environment “linking” / “unlinking” of packages (which are also tar.gz with standard prefix layout).

Yes, it can be implemented that way, and we may do that yet, but we’re gonna try doing something smarter to support faster switching of environments. None of this solves the system-provided library issue though.

Just wanted to add that if there was anyone from the Julia team who could help with the conda-forge recipe for Julia, that would be very useful.

A solid conda build of Julia would really help quick iteration on building bindings of other scientific software and Julia. (For example gdal, which is notoriously hard to build, is packaged on conda-forge, which would help for gdal.jl, same for bindings built upon cxxwrap, xtensor, and friends).

1 Like

Current build scripts seem to have a line like this:

const prefix = Prefix(!isempty(ARGS) ? ARGS[1] : joinpath(@__DIR__,"usr"))

When I run the pkg build command on such a package, there are no arguments, so the prefix remains inside the package directory. Is there already a way to change that so a common prefix can be used?

You’d probably have to change the package manager code (or you can just change the build.jl yourself of course). The reason they accept an argument is so BinaryBuilder can read them back in (so it would be good to preserve that if you change the default behavior). It’s unclear whether future iterations of this design will still use build.jl

To certain extent I agree with you. I’m currently maintaining julia for Debian, and I had been thinking whether to package things like JSON.jl for Debian. However I don’t see too much necessity of doing so. Besides, It looks not easy to package them since the UUID mechanism has been introduced. In the end I decided to not package any external .jl packages.

If you want to package Julia packages, you’d probably want to have a Debian-specific depot where only one of a given Julia package is ever installed. That would work fine with Pkg3 (much better than with other language package managers, in fact), since it would use those that are available while still allowing you to install and use other package from Julia.

Thanks for the pointer. I’m not quite familiar with Pkg3 and I’m still confused on how to package, for instance JSON.jl. May I ask for more hints?

The first impression is that I should create a depot directory layout under /usr/share/julia according to [1][2] and DEPOT_PATH. But then I didn’t figure out how to calculate e.g. HDkr in the path packages/Priv/HDkr/src/Priv.jl.

I tried to empty!(DEPOT_PATH); push!(DEPOT_PATH, somewhereelse), and then pkg> add file://xxx.jl . But Pkg3 still tries to clone the Registry. Network access is not allowed to build a package as per Debian policy.

Is there another (manual) way to install julia package to /usr/share/julia that doesn’t require network access? Or how to calculate HDkr?

Thanks :slight_smile:

[1] Code Loading · The Julia Language
I also found a document bug there: incorrect path – missing share, e.g. /usr/local/julia/packages/Priv/HDkr/src/Priv.jl
[2] julia/stdlib/Pkg/docs/src/