Right now, as I understand it,
Pkg.build is basically only called once when a package is installed; and again (?) whenever it is updated. That’s of course sufficient in basic use cases.
But I am dealing with multiple more complex packages where the reality is that at least a little bit of C/C++ glue code has to be compiled in
Pkg.build (BinaryBuilders are nice, but do not (yet?) cover all cases., sorry). And in those cases, the current conventions for Pkg.build are insufficient, it seems. E.g.:
Pkg.builderrored out, it is (AFAIK) still not called again, leaving the package potentially in a semi-broken state; we try to handle this by adding checks that refuse to load, but this has to be repeated again and again…
- … and to handle this, we face the choice of either telling the user to call
Pkg.build()or else to call it for them (the former gives more control to the user; the latter is “more convenient” when it works; but both are limited, e.g. calling
Pkg.build("Foobar")won’t work if
Foobaris not in the user’s environment, I think… this is a hack, in any case).
- some of the glue needs to link against Julia itself (specifically to access the GC code), and the resulting binaries are in general not exchangeable between Julia 1.x and 1.y; i.e. if built against Julia 1.3 it won’t work under 1.4 and vice versa. So if the user installs the package under 1.3, it won’t work under 1.4 unless they call Pkg.build there; after which it won’t work in 1.3, unless one takes special care (in one package, I simply insert the julia version into the name of the generated binary, so that I can have multiple versions of the generated shared libraries in parallel).
- this also affects users of CxxWrap: right now there is CxxWrap 0.9.1 and 0.10.1; they are not binary compatible. So either my package allows for either 0.9 or 0.10 only; or if I want to support both in
compat, then I need to take special care (see above; e.g. encode the CxxWrap / jlcxx version into the name of the generated binaries; or otherwise recording the version that was used to catch errors where code generated for one is accidentally used with the other).
These issues are admittedly rather special and probably not relevant for >99% of all packages. But I am also pretty sure we are not the only ones to be affected by it.
People have told me that I should just use BinaryBuilders and all problems go away, but I don’t see how; for starters, I can’t link against Julia in a BinaryBuilder (unless I try to use the Julia binary builder, which however is stuck at Julia 1.0 which is useless if (a) you need to access GC apis that were added later, and (b) when you consider that I said above that a compiled against 1.3 might not work with 1.4 and vice versa). Also, not sure how that would work with the CxxWrap situation I mentioned either (I’d need separate binary builders / binary builder version for each of CxxWrap 0.9 and 0.10…?)
I am not sure whether I can formulate a precise question at this point… I guess I wonder in general how to take care of these things, now, or in a hypothetical feature (as in: what kind of tools and feature could one add to help with this). I imagine it would already go a long way if I had a way to specify that my package needs to be rebuilt / is not properly built. Be it in a declarative manner (i.e. being able to specify: “this package needs to be rebuild if the Julia version changes; or if the version of this and that package changes”), or via a callback or whatever…
Also, perhaps I am overlooking some possibilities that exist right now?
One thing is that I could of course say we choose one, either CxxWrap 0.9 or 0.10, don’t try to support both at the same time. I actually kinda argued for this, but people are apparently of the opinion that this is OK, because our packages strictly speaking build with both; it’s “just” the resulting binaries that are incompatible. If you tell me “this is a stupid idea”, I am also happy to relay that