Could/should Pkg.build be triggered more often

Internals & Design
1

Right now, as I understand it, Pkg.build is basically only called once when a package is installed; and again (?) whenever it is updated. That’s of course sufficient in basic use cases.

But I am dealing with multiple more complex packages where the reality is that at least a little bit of C/C++ glue code has to be compiled in Pkg.build (BinaryBuilders are nice, but do not (yet?) cover all cases., sorry). And in those cases, the current conventions for Pkg.build are insufficient, it seems. E.g.:

  • if Pkg.build errored out, it is (AFAIK) still not called again, leaving the package potentially in a semi-broken state; we try to handle this by adding checks that refuse to load, but this has to be repeated again and again…
  • … and to handle this, we face the choice of either telling the user to call Pkg.build() or else to call it for them (the former gives more control to the user; the latter is “more convenient” when it works; but both are limited, e.g. calling Pkg.build("Foobar") won’t work if Foobar is not in the user’s environment, I think… this is a hack, in any case).
  • some of the glue needs to link against Julia itself (specifically to access the GC code), and the resulting binaries are in general not exchangeable between Julia 1.x and 1.y; i.e. if built against Julia 1.3 it won’t work under 1.4 and vice versa. So if the user installs the package under 1.3, it won’t work under 1.4 unless they call Pkg.build there; after which it won’t work in 1.3, unless one takes special care (in one package, I simply insert the julia version into the name of the generated binary, so that I can have multiple versions of the generated shared libraries in parallel).
  • this also affects users of CxxWrap: right now there is CxxWrap 0.9.1 and 0.10.1; they are not binary compatible. So either my package allows for either 0.9 or 0.10 only; or if I want to support both in compat, then I need to take special care (see above; e.g. encode the CxxWrap / jlcxx version into the name of the generated binaries; or otherwise recording the version that was used to catch errors where code generated for one is accidentally used with the other).

These issues are admittedly rather special and probably not relevant for >99% of all packages. But I am also pretty sure we are not the only ones to be affected by it.

People have told me that I should just use BinaryBuilders and all problems go away, but I don’t see how; for starters, I can’t link against Julia in a BinaryBuilder (unless I try to use the Julia binary builder, which however is stuck at Julia 1.0 which is useless if (a) you need to access GC apis that were added later, and (b) when you consider that I said above that a compiled against 1.3 might not work with 1.4 and vice versa). Also, not sure how that would work with the CxxWrap situation I mentioned either (I’d need separate binary builders / binary builder version for each of CxxWrap 0.9 and 0.10…?)

I am not sure whether I can formulate a precise question at this point… I guess I wonder in general how to take care of these things, now, or in a hypothetical feature (as in: what kind of tools and feature could one add to help with this). I imagine it would already go a long way if I had a way to specify that my package needs to be rebuilt / is not properly built. Be it in a declarative manner (i.e. being able to specify: “this package needs to be rebuild if the Julia version changes; or if the version of this and that package changes”), or via a callback or whatever…

Also, perhaps I am overlooking some possibilities that exist right now?

One thing is that I could of course say we choose one, either CxxWrap 0.9 or 0.10, don’t try to support both at the same time. I actually kinda argued for this, but people are apparently of the opinion that this is OK, because our packages strictly speaking build with both; it’s “just” the resulting binaries that are incompatible. If you tell me “this is a stupid idea”, I am also happy to relay that :wink:

2 Likes
2

I’ve run into this kind of issue also, using a BinaryBuilder-built library that uses CxxWrap (but does not yet use a JLL package, which is the newer way to use BinaryBuilder): Error when `using SDPAFamily` · Issue #29 · ericphanson/SDPAFamily.jl · GitHub. I am not really sure if building more is the solution though; it seems better if instead things (i.e. CxxWrap) can be changed so that Pkg.build does not depend on the Julia version. But I don’t know anything about how that works; maybe it’s not so easy.

edit: [julia]: Update build script to be compatible with new BB by jstrube · Pull Request #314 · JuliaPackaging/Yggdrasil · GitHub is related I think

3

Not an answer to your question, but have you looked at using BinaryBuilder to avoid the need to build on the client at all?

1 Like
4

@StefanKarpinski Yes, I tried (as mentioned in my post). Unfortunately it’s not possible in this case (at least in the current setup / design), because we need to link against Julia, and not just any, but precisely the Julia version the user runs; also CxxWrap needs access. See also Creating a binary that links against Julia · Issue #511 · JuliaPackaging/BinaryBuilder.jl · GitHub

5

Oh and I just discovered [julia] julia binaries for BB packages that need them as dependencies · Issue #321 · JuliaPackaging/Yggdrasil · GitHub which is also somewhat related

6

Maybe you can compile the C/C++ code during precompilation of the Julia package? At this point, the versions of Julia and the dependencies (e.g., CxxWrap) are known so you can create different files for them.

7

We currently at least store the CxxWrap version that was used to build, so we can catch the “mixup” situation and at least show a nice error, instead of segfaulting. And for another package (which doesn’t use CxxWrap but still faces a similar problem), I put the version of the dependency that cause the problem (here: Julia itself!) into the path where the compiled files are put; this way, I can have multiple copies of the compiled code, one for each distinct variant.

But note that the limitations of Pkg.build go further: e.g. if a package build fails temporarily (e.g. hard disk is full; some external dep was missing; the user hard pressed Ctrl-C; …), then after resolving the cause for the temporary failure, the package is left in a semi-broken state, which a user has to know how to resolve. Or packages have to implement a system to track if they were really built successfully (I did that for one of mine). I can of course start doing that for all packages I control or influence, but to me this seems like an issue that needs to be addressed on a wider scaler; otherwise I am sure many packages that would benefit from these safety nets won’t have them, simply because their authors are not aware of this need to manually catch and guard for various error situations.

But Julia resp. Pkg.jl could help with this:

  • they could track whether a package build actually was successfully (of course this also requires a way for package build systems to signal this)
  • Pkg.status could show if a package was not built yet / had a build error
  • there could be a way for a package to signal that it may need to be rebuilt “more often” than right now; this could even be purely declarative: say via a new entry type in Project.toml: a section [trigger_rebuild] which list package names (plus “Julia”) which, if updated, require a rebuild of the package; this could also specify what kinds of updates trigger an update:
[trigger_rebuild]
julia = minor   # trigger rebuild on minor updates, e.g. 1.3 -> 1.4; but not for patch release 1.3 -> 1.3.1
CxxWrap = major # 0.9 to 0.10 triggers, but not 0.9 to 0.9.1.
Foobar = all # any version change of Foobar.jl triggers a rebuild

to implement this, one could record for the package:

  • whether it was built successfully
  • and if so, the exact version of each trigger_rebuild dependency when the build was done.

Of course this won’t cover every single possible cases, but I think it’d cover a majority of them (at least all of mine), would fit well into the current setup, and should be relatively easy to implement (I mean, of course still might involve considerable amounts of work, but doesn’t seem to pose any serious challenges in terms of design that need to be resolved – well, except of course for all those I am missing :wink: so I’ll wait for people here to punch holes into my idea).

3 Likes
8

By the way, is there any way a package can “build itself” ? I tried Pkg.build("foobar") but that only works if "foobar" is added in the current environment. I also tried variations of Pkg.build(PKGDIR), or using the UUID, etc. but none work.

9

Ref Build failures should probably be more fatal · Issue #1411 · JuliaLang/Pkg.jl · GitHub

1 Like
10

To expand on my last question: we had several reports which traced down to build.jl failing for a package because a download failed intermittently (perhaps their network connection was flaky; perhaps it was during GitHub’s recent outage; I don’t know). Of course artifacts greatly help with detecting partial/corrupt downloads, but they don’t help me restart the interrupt build.

So we are now trying hard in our packages to detect if Pkg.build failed, and tell the user to re-run it. Unfortunately, if the user installed package A, which depends on package B, and B had a failure, the only thing it can tell the user is to run using Pkg; Pkg.build("B") – but that might fail because the user only added A, not B, to their environment; so I really need to suggest to users to do using Pkg; Pkg.add("B"); Pkg.build("B"). This is IMHO not very user friendly.

Are there any alternatives to this that I am missing?

11

Pkg.build("A") triggers building of its deps too, at least that is the case for me.

1 Like
12

True. But the error message produced by package B can’t tell the user that, because package B doesn’t know that it was pulled in as a dependency to A.