Hi all,
I’d like to automatically upload binaries from an Appveyor test. For a release, binaries can be linked to a release on Github, but where can the binaries be stored for the latest master version? Ideally, I would like to obtain the following workflow:
The users of your julia package don’t really need this CI binary building infrastructure to get installed on their systems, so I’d split it into a separate repo from the Julia package.
Appveyor lets you upload artifacts from any given build, but I don’t think they’ll be available indefinitely. Github releases lets you attach files, but only to a release tag (or probably to a pre release?). You could also use a different service such as bintray that allows programmatic uploading through an api.
And expanding on what @tkelman said with some of his own advise from another spot on the web:
https://github.com/JuliaDiffEq/Sundials.jl/pull/89#issuecomment-268795386
Given this I am going to be taking a crack at using Opensuse build service + WinRPM, but whatever way you find works out for you please share!
For CxxWrap, this now works with AppVeyor, using this script. The additions needed start at line 44, where the deps/usr directory is zipped. The artifacts section makes sure the zip file is available for download and the deploy section uploads the binaries to a github release when the build is happening on a tag. With these fairly simple changes, unreleased master binaries are directly available from appveyor, using a permalink that is based on the build name. Release binaries appear automatically as attachments to the GitHub release when a tag is pushed. Combined with @simonbyrne 's excellent Attobot, this means making a release with binaries is as simple as creating the release from the GitHub web interface. Even just pushing a tag from the command line would work, since AppVeyor will create a release for the tag automatically.
The major caveat is that AppVeyor doesn’t have a compiler that matches the Julia compiler, this is discussed at length here. The duration of the compilation process is also limited by the AppVeyor limits, which may be a problem for larger packages.
It makes no sense at all to be downloading a binary that hasn’t been built yet when you tag your package. For one thing you should never download a file that doesn’t get checksum-verified on the user’s system. And using Pkg.installed is not going to work for a package installed elsewhere on LOAD_PATH.
This should not happen: the binaries are uploaded mere minutes after the tag is created, so before the release is in metadata. If this becomes a problem when metadata gets decentralized, it should be doable to adapt attobot to postpone the pull request to metadata until the binaries are in place.
I wasn’t using checksums before, either. Maybe that’s wrong, but I also don’t see how it would add security: the checksum is provided on the same site as the binaries (GitHub) and even if I build on my own machine, how do I know if it’s not infected with a virus? Also wouldn’t the checksum have to be cryptographically signed to be of real use for security purposes?
I would be happy to learn how to correctly query the package version, then. I looked at the Libgit2 package, but couldn’t figure out how to get the tag from there.
Do you build non-tagged commits on appveyor, or only tags? If you introduce a bug that causes the build to fail, will you notice before or after you make a release tag? I don’t think you should be tying the file name that you download to the version number of the Julia code. The binary only contains compiled C/C++ code, not the rest of the contents of the package, and can change at a different pace.
Without a checksum to sanity-check the contents of the file, it could contain anything - nothing prevents you, or anyone else with commit access, from taking down the original binaries and replacing them with anything else, which is bad for both security and reproducibility. A checksum would at least provide download integrity verification and verify that the contents match what you originally intended, and if your build process is reproducible then it could be repeated by an independent party and arrive at binary-identical contents.
I build both, so I only tag if the last commit passed the tests. In practice, that means the same binaries are generated a second time when AppVeyor picks up the tag.
Well, in the (presumably exceptional) case of CxxWrap and its dependents, most of the work is actually in the C++ code and most changes require new binaries. But in any case, the github releases are tied to the tag name, so I need the version number for the URL.
True, I suppose I could include checksums based on the successful test run that precedes the tagging and then commit those to a source file in the deps directory. That way the checksums are tied to the git history and there is no way to modify them secretly. Maybe it can even be automated, I was thinking committing back from the build would set off a destructive AppVeyor chain reaction, but it seems to be supported.
Maybe an even better place would be to have the checksums in metadata? Is this something that could be considered for the new Pkg version? If it’s there, attobot could take care of the checksumming.
I wouldn’t commit binaries either, git would not handle that well and it would inflate your repo size for users on all platforms with files that aren’t always relevant. This is why I suggested a separate repository as a more appropriate place to do your CI binary building and uploading from, still using github releases as the upload location. If you don’t want to decouple the Julia source from the C/C++ source, then the builder repo could just have all of CxxWrap as a submodule and only use the C/C++ source part. Your users don’t need the CI auto-build scaffolding so they won’t have to worry about submodules’ usability issues. And the auto-build parts of your CI on the Julia package aren’t useful for representative testing of installation of the package on users’ machines who don’t have compilers installed. Tags and rebuilds of the binaries can be versioned separately from the Julia package tags. A Julia package tag should download a specific checksummed version of the binaries, but the versions don’t necessarily have to increment in a one-to-one fashion.
Yes, I wasn’t suggesting putting the binaries in the git repo, just auto-committing the checksums on building. I did consider separating the C++ from the Julia code, but in the end it was too impractical: because I also link back to the Julia C API, the binaries themselves depend on the Julia version and finding the correct library is most easily managed from within the build.jl. The current setup also makes it easy to contribute: users just need to modify a file in the package source tree and call Pkg.build(), and things will just work even if it was a C++ file that got changed (or throw a horrible compiler error on even the slightest of mistakes, of course). There is not much overhead from the auto-build parts, most of the code used for building is also used to build the package when it is installed through Pkg.add on Linux or macOS. Finally, moving the binaries to a separate repo feels a bit awkward because they don’t actually work at all without the Julia part of the package.
I see now the situation of CxxWrap is a bit particular however, the separate build repo could be a good solution, I’ll have to read up on submodules a bit to see how it would fit together.