Binary dependencies

[SylvainCorlay]

@Keno

@SylvainCorlay

If there was a pure Julia client to conda channels, that would probably be a much more scalable thing to adopt, and much closer to the model of linux distributions.

So you’re advocating that we adopt conda-forge, and don’t support any other package manager (e.g. the HPC center use case suggested above)? If you’re not advocating that we only adopt conda-forge, then we’re back to the exact same situation, where we somehow need to identify what each library is called in each package manager and where to find it. There’s a separate discussion on conda-forge vs BinaryBuilder, which we can certainly have, but it’s some tangential to this discussion.

I was pointing at conda-forge (which is ~2yo) as an example that language-agnostic approach is more likely to scale, and would help making Julia more inter-operable with other large code bases.

I am advocating

• for the official download bundle Julia to ship a simple installation prefix including Julia as a package installed among others.
• that this installation prefix may be used to install other assets.
• that whatever underlies Pkg.add installs non-julia dependencies as normal packages in that installation prefix.

[Keno]

I think we’re talking past each other here. What I’m addressing is how to support binary dependencies installed by the system package manager (or an HPC administrator) and re-using them from within julia.

@SylvainCorlay

I am advocating

• for the official download bundle Julia to ship a simple installation prefix including Julia as a package installed among others.
• that this installation prefix may be used to install other assets.
• that whatever underlies Pkg.add installs non-julia dependencies as normal packages in that installation prefix.

This is how things work with BinaryBuilder (the actual binary packages are just .tar.gz files with standard prefix layout), with the additional complication that the prefix is sharded to allow for quick switching between versions (of course you could implement this the other way, with having a single prefix that gets put together every time you activate an environment, but that’s an implementation detail)

[SylvainCorlay]

@Keno

This is how things work with BinaryBuilder (the actual binary packages are just .tar.gz files with standard prefix layout), with the additional complication that the prefix is sharded to allow for quick switching between versions (of course you could implement this the other way, with having a single prefix that gets put together every time you activate an environment, but that’s an implementation detail)

Ok, this sounds like the conda environment “linking” / “unlinking” of packages (which are also tar.gz with standard prefix layout).

[Keno]

@SylvainCorlay

Ok, this sounds like the conda environment “linking” / “unlinking” of packages (which are also tar.gz with standard prefix layout).

Yes, it can be implemented that way, and we may do that yet, but we’re gonna try doing something smarter to support faster switching of environments. None of this solves the system-provided library issue though.

[SylvainCorlay]

Just wanted to add that if there was anyone from the Julia team who could help with the conda-forge recipe for Julia https://github.com/conda-forge/julia-feedstock, that would be very useful.

A solid conda build of Julia would really help quick iteration on building bindings of other scientific software and Julia. (For example gdal, which is notoriously hard to build, is packaged on conda-forge, which would help for gdal.jl, same for bindings built upon cxxwrap, xtensor, and friends).

[barche]

@Keno

The rough plan is to not have a prefix at all (and have the julia dynamic library loader resolve the dependencies appropriately) or have a single prefix per environment that installs the correct versions of things,

Current build scripts seem to have a line like this:

const prefix = Prefix(!isempty(ARGS) ? ARGS[1] : joinpath(@__DIR__,"usr"))

When I run the pkg build command on such a package, there are no arguments, so the prefix remains inside the package directory. Is there already a way to change that so a common prefix can be used?

[Keno]

@barche

When I run the pkg build command on such a package, there are no arguments, so the prefix remains inside the package directory. Is there already a way to change that so a common prefix can be used?

You’d probably have to change the package manager code (or you can just change the build.jl yourself of course). The reason they accept an argument is so BinaryBuilder can read them back in (so it would be good to preserve that if you change the default behavior). It’s unclear whether future iterations of this design will still use build.jl

[CDLuminate]

@nalimilan

But that can only be done on a case by case basis after distribution developers have reviewed these libraries and checked that everything works. And if they are outdated Julia packages have to download an up-to-date version.

To certain extent I agree with you. I’m currently maintaining julia for Debian, and I had been thinking whether to package things like JSON.jl for Debian. However I don’t see too much necessity of doing so. Besides, It looks not easy to package them since the UUID mechanism has been introduced. In the end I decided to not package any external .jl packages.

[StefanKarpinski]

If you want to package Julia packages, you’d probably want to have a Debian-specific depot where only one of a given Julia package is ever installed. That would work fine with Pkg3 (much better than with other language package managers, in fact), since it would use those that are available while still allowing you to install and use other package from Julia.

[CDLuminate]

Thanks for the pointer. I’m not quite familiar with Pkg3 and I’m still confused on how to package, for instance JSON.jl. May I ask for more hints?

The first impression is that I should create a depot directory layout under /usr/share/julia according to [1][2] and DEPOT_PATH. But then I didn’t figure out how to calculate e.g. HDkr in the path packages/Priv/HDkr/src/Priv.jl.

I tried to empty!(DEPOT_PATH); push!(DEPOT_PATH, somewhereelse), and then pkg> add file://xxx.jl . But Pkg3 still tries to clone the Registry. Network access is not allowed to build a package as per Debian policy.

Is there another (manual) way to install julia package to /usr/share/julia that doesn’t require network access? Or how to calculate HDkr?

Thanks

[1] Code Loading · The Julia Language
I also found a document bug there: incorrect path – missing share, e.g. /usr/local/julia/packages/Priv/HDkr/src/Priv.jl
[2] julia/stdlib/Pkg/docs/src/index.md