# Binary dependencies

**URL:** <https://discourse.julialang.org/t/binary-dependencies/11623>\
**Category:** Internals & Design\
**Created:** [June 12, 2018, 8:46pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623 "2018-06-12T20:46:52Z")\
**Posts on this page:** 10\
**Page:** 2

<div class="post-metadata">

**Author:** ![SylvainCorlay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaincorlay/32/923_2.png) [@SylvainCorlay](https://discourse.julialang.org/u/SylvainCorlay)\
**Post date:** [June 13, 2018, 4:01pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/21 "2018-06-13T16:01:46Z")

</div>

> [@Keno](#):
>
> > [@SylvainCorlay](#):
> >
> > If there was a pure Julia client to conda channels, that would probably be a much more scalable thing to adopt, and much closer to the model of linux distributions.
> 
> So you’re advocating that we adopt conda-forge, and don’t support any other package manager (e.g. the HPC center use case suggested above)? If you’re not advocating that we only adopt conda-forge, then we’re back to the exact same situation, where we somehow need to identify what each library is called in each package manager and where to find it. There’s a separate discussion on conda-forge vs BinaryBuilder, which we can certainly have, but it’s some tangential to this discussion.

I was pointing at conda-forge (which is ~2yo) as an example that language-agnostic approach is more likely to scale, and would help making Julia more inter-operable with other large code bases.

I am advocating

- for the official download bundle Julia to ship a simple installation prefix including Julia as a package installed among others.
- that this installation prefix may be used to install other assets.
- that whatever underlies `Pkg.add` installs non-julia dependencies as normal packages in that installation prefix.

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [June 13, 2018, 4:05pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/22 "2018-06-13T16:05:32Z")

</div>

I think we’re talking past each other here. What I’m addressing is how to support binary dependencies installed by the system package manager (or an HPC administrator) and re-using them from within julia.

> [@SylvainCorlay](#):
>
> I am advocating
> 
> - for the official download bundle Julia to ship a simple installation prefix including Julia as a package installed among others.
> - that this installation prefix may be used to install other assets.
> - that whatever underlies `Pkg.add` installs non-julia dependencies as normal packages in that installation prefix.

This is how things work with BinaryBuilder (the actual binary packages are just .tar.gz files with standard prefix layout), with the additional complication that the prefix is sharded to allow for quick switching between versions (of course you could implement this the other way, with having a single prefix that gets put together every time you activate an environment, but that’s an implementation detail)

---

<div class="post-metadata">

**Author:** ![SylvainCorlay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaincorlay/32/923_2.png) [@SylvainCorlay](https://discourse.julialang.org/u/SylvainCorlay)\
**Post date:** [June 13, 2018, 4:13pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/23 "2018-06-13T16:13:10Z")

</div>

> [@Keno](#):
>
> This is how things work with BinaryBuilder (the actual binary packages are just .tar.gz files with standard prefix layout), with the additional complication that the prefix is sharded to allow for quick switching between versions (of course you could implement this the other way, with having a single prefix that gets put together every time you activate an environment, but that’s an implementation detail)

Ok, this sounds like the conda environment “linking” / “unlinking” of packages (which are also tar.gz with standard prefix layout).

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [June 13, 2018, 4:14pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/24 "2018-06-13T16:14:42Z")

</div>

> [@SylvainCorlay](#):
>
> Ok, this sounds like the conda environment “linking” / “unlinking” of packages (which are also tar.gz with standard prefix layout).

Yes, it can be implemented that way, and we may do that yet, but we’re gonna try doing something smarter to support faster switching of environments. None of this solves the system-provided library issue though.

---

<div class="post-metadata">

**Author:** ![SylvainCorlay](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sylvaincorlay/32/923_2.png) [@SylvainCorlay](https://discourse.julialang.org/u/SylvainCorlay)\
**Post date:** [June 20, 2018, 6:10pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/25 "2018-06-20T18:10:56Z")

</div>

Just wanted to add that if there was anyone from the Julia team who could help with the `conda-forge` recipe for Julia [https://github.com/conda-forge/julia-feedstock](https://github.com/conda-forge/julia-feedstock), that would be very useful.

A solid conda build of Julia would really help quick iteration on building bindings of other scientific software and Julia. (For example gdal, which is notoriously hard to build, is packaged on conda-forge, which would help for gdal.jl, same for bindings built upon cxxwrap, xtensor, and friends).

---

<div class="post-metadata">

**Author:** ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)\
**Post date:** [June 22, 2018, 9:59pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/26 "2018-06-22T21:59:58Z")

</div>

> [@Keno](#):
>
> The rough plan is to not have a prefix at all (and have the julia dynamic library loader resolve the dependencies appropriately) or have a single prefix per environment that installs the correct versions of things,

Current build scripts seem to have a line like this:

```julia
const prefix = Prefix(!isempty(ARGS) ? ARGS[1] : joinpath(@ __DIR__ ,"usr"))

```

When I run the pkg `build` command on such a package, there are no arguments, so the prefix remains inside the package directory. Is there already a way to change that so a common prefix can be used?

---

<div class="post-metadata">

**Author:** ![Keno](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/keno/32/285_2.png) [@Keno](https://discourse.julialang.org/u/Keno)\
**Post date:** [June 22, 2018, 10:19pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/27 "2018-06-22T22:19:36Z")

</div>

> [@barche](#):
>
> When I run the pkg `build` command on such a package, there are no arguments, so the prefix remains inside the package directory. Is there already a way to change that so a common prefix can be used?

You’d probably have to change the package manager code (or you can just change the build.jl yourself of course). The reason they accept an argument is so BinaryBuilder can read them back in (so it would be good to preserve that if you change the default behavior). It’s unclear whether future iterations of this design will still use build.jl

---

<div class="post-metadata">

**Author:** ![CDLuminate](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdluminate/32/6787_2.png) [@CDLuminate](https://discourse.julialang.org/u/CDLuminate)\
**Post date:** [September 26, 2018, 11:38am UTC](https://discourse.julialang.org/t/binary-dependencies/11623/28 "2018-09-26T11:38:20Z")

</div>

> [@nalimilan](#):
>
> But that can only be done on a case by case basis after distribution developers have reviewed these libraries and checked that everything works. And if they are outdated Julia packages have to download an up-to-date version.

To certain extent I agree with you. I’m currently maintaining julia for Debian, and I had been thinking whether to package things like `JSON.jl` for Debian. However I don’t see too much necessity of doing so. Besides, It looks not easy to package them since the UUID mechanism has been introduced. In the end I decided to not package any external .jl packages.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [September 26, 2018, 8:30pm UTC](https://discourse.julialang.org/t/binary-dependencies/11623/29 "2018-09-26T20:30:21Z")

</div>

If you want to package Julia packages, you’d probably want to have a Debian-specific depot where only one of a given Julia package is ever installed. That would work fine with Pkg3 (much better than with other language package managers, in fact), since it would use those that are available while still allowing you to install and use other package from Julia.

---

<div class="post-metadata">

**Author:** ![CDLuminate](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cdluminate/32/6787_2.png) [@CDLuminate](https://discourse.julialang.org/u/CDLuminate)\
**Post date:** [September 27, 2018, 2:27am UTC](https://discourse.julialang.org/t/binary-dependencies/11623/30 "2018-09-27T02:27:13Z")

</div>

Thanks for the pointer. I’m not quite familiar with Pkg3 and I’m still confused on how to package, for instance JSON.jl. May I ask for more hints?

The first impression is that I should create a depot directory layout under `/usr/share/julia` according to [1][2] and DEPOT\_PATH. But then I didn’t figure out how to calculate e.g. `HDkr` in the path `packages/Priv/HDkr/src/Priv.jl`.

I tried to `empty!(DEPOT_PATH); push!(DEPOT_PATH, somewhereelse)`, and then `pkg> add file://xxx.jl` . But Pkg3 still tries to clone the Registry. Network access is not allowed to build a package as per Debian policy.

Is there another (manual) way to install julia package to `/usr/share/julia` that doesn’t require network access? Or how to calculate `HDkr`?

Thanks 🙂

[1] [Code Loading · The Julia Language](https://docs.julialang.org/en/v1.0.0/manual/code-loading/)  
I also found a document bug there: incorrect path – missing `share`, e.g. `/usr/local/julia/packages/Priv/HDkr/src/Priv.jl`  
[2] julia/stdlib/Pkg/docs/src/index.md

[Previous page](https://discourse.julialang.org/t/binary-dependencies/11623.md?page=1)