Windows C++ binary compatibility: MinGW vs MSVC

General Usage
,
1

Creating a new topic to continue the discussion about binary and compiler compatibility started here in the context of QML.jl.

All of this is in the context of CxxWrap.jl, where C++ code is made available in Julia by building a small C++ wrapper library that is accessed from Julia using ccall. CxxWrap itself contains a C++ library, and so does any package built using it. Naturally, all of these have to be compiled with the same compiler for any binaries to be compatible with each other. Furthermore, in the event somene else but me would want to release Windows binaries for a CxxWrap-based package, this author would need to release his binaries built with the same compiler as I used for the CxxWrap binary release. The wrapped library itself (e.g. parts of Qt in case of QML.jl) also must be built with that same compiler.

Using Appveyor, it is possible to build binaries automatically using either MinGW or Visual Studio. Since Julia itself is build using Mingw, using that seems most logical, as suggested by @tkelman

The binary distribtion of Julia itself also contains a libstdc++ dll, however. Building with Mingw results in binaries that require the libstdc++ of the mingw compiler, which is different from the one used to compile Julia. For now, this seems to result in errors only on the 32 bit Windows version. Linking CxxWrap statically with libstdc++ solves it for CxxWrap, but for QML this doesn’t work.

Using MSVC, the C++ standard library is in an entirely different DLL, so the conflict doesn’t exist and both 64bit and 32bit binaries work fine for QML and CxxWrap. There is a concern when mixing Mingw and MSVC in the case of allocating an object in an MSVC dll and deallocating in a Mingw dll or vice versa, however I believe this is no issue here, since calls from Julia into CxxWrap happen only with ccall, and calls from CxxWrap into Julia happen only using the official Julia C interface, so allocation and deallocation always happen on the same side. This is also confirmed by the fact that all tests pass using MSVC.

In conclusion, I see 3 solutions:

  1. Use the standard Mingw compiler on Appveyor and abandon binaries on the 32 bit Windows platform
  2. Mix Mingw Julia with MSVC CxxWrap and QML binaries
  3. Use the exact same compiler as the one used for Julia

While option 3 is best theoretically, it has two major practical obstacles for which I don’t see the solution:

  1. How do I know which compiler to use, and where can I get it (automatically in an appveyor script)? Also, does that compiler risk changing between minor Julia revisions?
  2. How do I get Qt binaries built with that compiler (I don’t see building Qt myself as an option, automating it on Appveyor will also never fit in the time limit)?
1 Like
2

Which mingw is getting used on appveyor? There are multiple variants. You
should absolutely avoid the 32 bit mingw.org variant as it’s obsolete. The
i686 build of mingw-w64 is much more capable and what Julia uses on win32.
Julia binaries are cross compiled using the cygwin package for mingw-w64
gcc 4.9.2.

You can also use WinRPM which has cross-compiled mingw-w64 binaries of qt
and many other libraries already available from the opensuse build service.
The compiler used on opensuse should be binary compatible with the one used
to build Julia itself. If you find a case where it isn’t then we can work
to fix that.

3

It’s the MSYS2 version, which is indeed the w64 variant but it is at version 5.3 in the case of Appveyor.

OK, is there an easy way or example script on how to get these onto Appveyor? I just looked at WinRPM.jl, but if I install gcc using that I get version 6.3.

4

As a far note, I still hope that one day Julia on Windows will be build with MSVC, and from (very modest) side I have no intention to build my dependencies with anything else but that compiler.

5

Do you control the C/C++ code? If you do, it should generally be possible to avoid problematic function signatures.

6

MSYS2 uses a different exception handling variant on 32 bit than every
other common mingw-w64 build. You’ll notice that in complicated enough C++
code like qt. That’s the main reason we don’t really support MSYS2 in a
first class way.

Easiest way is via the Julia package. There are a few examples of packages
that have done this, notably ParallelAccelerator. It’s doable without
installing Julia too in a few ways, but messier (see
contrib/windows/get_toolchain.sh).

7

I couldn’t get CMake to play along with the WinRPM-provided GCC, which is version 6.2 anyway and so probably incompatible with Julia. Looking at the Julia build script, I did install the GCC 4.9.2 compiler from your bintray archive, and this installs CxxWrap cleanly (no need to link statically to libstdc++ or copy over extra DLLs, runs fine outside of the MinGW shell after compilation).

QML is a different matter, unfortunately:

  • The WinRPM binaries link against libstdc++ 6.2, but I couldn’t even test if it’s a problem because the development packages are broken and CMake bails out on this:
 The imported target "Qt5::Core" references the file

     "/usr/i686-w64-mingw32/bin/qt5/qmake"

  but this file does not exist.
  • The official Qt5 installer provides MinGW 4.9.2 binaries, as it happens, but linked DLL also fails to load, presumably because it uses a different MinGW variant (there is a libgcc_s_dw2-1.dll in the Qt installation, which is the wrong one I guess).

I do control part of the C++ code (at the interface between Julia and the external library), but here the problem is that incompatible versions of libstdc++ are needed at the same time. Since Julia loads its own libstdc++ at startup, any shared library that relies on another libstdc++ fails to load because its own libstdc++ never gets loaded since the Julia one is already in memory. At least that’s how I understand what happens :slight_smile:

For some reason (presumably the exception model) this is only a problem on 32bit Windows.

Unless compatible Qt binaries exist out there, I’m leaning towards the MSVC solution now.

8

The Qt binaries should work fine in cross-compilation. Opensuse uses them to cross compile several large Qt applications. Likely not from Windows, but Windows is an awful place to compile anything from anyway. If there’s zero requirement to build things the same place you run them, and they’re much easier to build somewhere else, should probably do that. MSVC is not going to be supported by Julia until an organization or individuals who want it to happen sponsor it financially or with code contributions.

9

Don’t use the bintray downloads for anything by the way. Those are used for appveyor tests but nothing else.

10

OK, is there an example script on how to do the cross-compilation then? Could it also be done from cygwin? That way I could use Appveyor (which has cygwin too) to build the binaries automatically upon committing and upload binaries to github upon tagging (this is related to the thread about uploading binaries, Appveyor works very well for that, I’ll update that topic once the correct binaries are built and I can test a complete release).

11

If you want to do cross-compilation from the opensuse build service to make binaries available via WinRPM, the closest thing to a tutorial is here: https://github.com/JuliaPackaging/WinRPM.jl/issues/28#issuecomment-121844977 - I never got around to properly cleaning that up and working through an example in full details though. This more or less works today, but I want to put together a better system that will make things more uniform across platforms and require less remote trial-and-error debugging through the build service (or CI services).

It does look like Cygwin has mingw-w64 cross-compiled qt binaries available as well: https://cygwin.com/cgi-bin2/package-grep.cgi?grep=mingw.*qt&arch=x86_64. These weren’t available in mainline cygwin when WinRPM was first written, but they are now. Worth a try. Though note that due to a bug in GCC 5, we’re still using an older version (4.9.2) of the mingw-w64 GCC cross-compiler from cygwin to build the binaries of Julia itself. The simplest way to get those today is going through Cygwin Time Machine Cygwin Time Machine from roughly September 2016. And cross-compiling anything from Cygwin is going to be much slower than cross-compiling from Linux.

12

OK, so I tried cygwin, but cmake immediately crashes with an error that the cygwin DLL is loaded at two different addresses. At this point, it seems that there is no easy, automated way to release Windows binaries that are built with the same compiler as Julia.

As a short-term solution, I propose to continue with the MSVC binaries for CxxWrap.jl and QML.jl. I believe because of the way CxxWrap works, the potential caveats for mixing the binaries don’t apply (it’s a pure C interface and exceptions and allocations don’t cross dll boundaries). It seems safer than mixing different MinGW versions, especially on 32 bit.

In the long term, I would rather direct my energy towards getting a working CMake build for Julia, which would be an important step towards getting an MSVC build. Is this still a good starting point for the CMake integration:
https://github.com/JuliaLang/julia/pull/11754
?

13

That sounds like AppVeyor’s installation of Cygwin is broken. Open an issue with them.

Using MSVC would mean imposing that every one of your users on Windows has to also build their libraries with a compiler that is not supported by Julia (or most of the open source ecosystem for that matter - need inline assembly or an autotools build system or Fortran or want to cross-compile? MSVC is useless), and that their libraries also avoid all the problems from mixing runtimes.

Yes, #11754 is the last I’ve heard of anyone making a serious attempt at converting Julia’s build system to cmake. To make it a fully-fledged source build including the dependencies would be even more complicated, see https://github.com/JuliaLang/julia/pull/7761#issuecomment-58848806 for an estimation of the various deps as of a few years ago.

14

This was a fresh install on my home machine, in fact. I would need too many iterations on Appveyor to try it there right away.

Yes, this is why I consider the MSVC solution a temporary stop-gap until I find the time to get something better. Note that CxxWrap only interfaces with the C++ part, which should usually compile with a recent MSVC compiler. Any external C or Fortran libraries can be built using whatever compiler works. On the other hand, there may be users who want to wrap a library that is developed mainly using MSVC, so it’s impossible to predict which choice is best for potential CxxWrap users. Until the problem presents itself, I might as well go with MSVC, it seems to me. If a concrete problem crops up for a package we can deal with it then. Right now, my concrete problem is that there is no simple solution for QML.jl (32 bit especially) except for MSVC. If you want, I can put a summary of this discussion in the CxxWrap readme to warn developers.

I understand none of this is ideal, but building QML.jl in cross-compile mode is also far from ideal from my point of view: it requires maintaining a completely different build script just for the windows binaries, and this would need to be run manually for every release, and no master branch binaries would be available. The result is then even sub-optimal, with only Qt 5.5 being available, which has some gaps (I haven’t checked how important) in windows 10 support and which lacks high-dpi support on Windows. Compare this to MSYS2 or MSVC, where either the very latest Qt or at least version 5.6 (an LTS release) is available, and all it takes to build a binary is a push to GitHub (all taken care of by a fairly simple AppVeyor script).

For now, I think using MSVC for CxxWrap and its dependents should be workable, there is also Conda (as already used by PyCall.jl) that can also be used as a source of MSVC-compatible binaries for hard-to-build dependencies. If an issue comes up, I’m willing to put in more effort to solve it then, but until there is a concrete case I would much rather focus on getting the next CxxWrap+QML release out (and the embryonic Trilinos.jl, which is actually relevant for my work, go figure ;).

15

This was a fresh install on my home machine, in fact.

Then your fresh installation of cygwin was also broken. Be sure you don’t have any junk on your path - especially not a separate installation of MSYS2.

a library that is developed mainly using MSVC

I’ve never seen a library that builds only with MSVC and can’t be made to work with GCC. That doesn’t mean it doesn’t exist, but interoperating between such a library and Julia is going to be problematic anyway.

it requires maintaining a completely different build script just for the windows binaries

For the subset of the open-source ecosystem that uses cmake and can function despite MSVC’s many limitations as a compiler, then maybe you’ll have more in common that way. To every other project out there, MSVC is way out of the norm - gcc at least behaves more or less the same no matter what OS you’re targeting. Take a look at how much of conda-forge doesn’t build for Windows at all because Python has inflicted MSVC on all its Windows users. No Julia packages that I’m aware of are using Conda-built MSVC binaries for ccall purposes.

16

I don’t think I put MSYS2 on the path (not consciously anyway), but it’s worth checking.

I’m not an expert at all on it, but Tensorflow seems to be an example, though I think the MSVC is imposed because of Python once again. My main point is that it’s impossible to predict wether MSVC or cross-compilation will be easiest for the next hypothetical CxxWrap user, there are a lot of domains in science and maybe the automake-based stack isn’t the dominant solution in all of them.

True, and again, I agree in principle, but the situation right now is that the only released CxxWrap-using package is QML.jl (as far as I’m aware, is there a way to check?), and it has working MSVC binaries on Windows (all tests passing until I started requiring an unreleased CxxWrap, even the GLVisualize integration works). So the pragmatic thing to do for now seems to be to just use these, and make time for a better fix when and if someone absolutely needs CxxWrap to be cross-compiled.

With the exception of PyCall.jl ccalling into the Python libraries then… One could equally argue that there a cross-compiled Python should be used instead. Just kidding of course, but the comparison is not that far fetched: Both PyCall and CxxWrap use a pure C interface to interact with the MSVC DLLs, both restrict their dependents to MSVC-compatible code. And for CxxWrap I’m willing to deal with that when and if the use case presents itself. Maybe by that time the MSVC clang integration will be a fact and the problem is resolved without any effort.

17

And gcc works fine with cmake. We’ll be putting together better tooling, documentation, and automation to simplify the process of cross-compiling and producing Windows binaries for package developers. MSVC is not going to be a part of this given its limitations. Initial porting of libraries like Tensorflow to add Windows support would likely have been easier if GCC were the target compiler rather than trying to support both a new platform and a new compiler / build system at the same time.

One could equally argue that there a cross-compiled Python should be used instead.

Yes. This would eliminate the need to deal with a bunch of MSVC-imposed issues like “activation contexts” and such, that it’s totally unreasonable to expect package developers who don’t even use Windows themselves to figure out. The only reason we don’t do this is that cpython’s developers refuse to review long-standing sets of patches that add support for building with GCC on Windows.

Maybe by that time the MSVC clang integration will be a fact and the problem is resolved without any effort.

Clang is unlikely to ever be a standalone toolchain. You can only use clang in combination with either the MSVC runtime or the MinGW runtime. While there are plans to update MinGW so it’s capable of linking against the recent UCRT runtime, the project that was working on that has completely stalled and made no progress for nearly a year.

18

Sounds promising. Would switching to msys2 be an option? It works nicely as I recently discovered, and the exception handling is only different on win32, which is a dying platform anyway?

To conclude regarding CxxWrap: are you willing to accept the MSVC binaries until the tooling is better and e.g. PyCall also makes the switch?

19

FYI, SymEngine.jl is using conda binaries for libsymengine which is a C++ library with manually written C wrappers and compiled using MSVC. (libsymengine is used in a python package and hence compiled using MSVC). In SymEngine.jl when we switched to using Conda, I tried an appveyor.yml and it worked, and it didn’t occur to me that Julia is using MinGW and there might be problems. If I had known, I would have used MinGW, but now that the MSVC binaries have been used for some time, and haven’t had any mysterious segfaults, I’m fine with it.

20

Not unless Julia itself were built with their toolchain, which we don’t currently support. As with MSVC (but likely much easier) - if someone wants to package Julia in MSYS2 you can go ahead and do that, then all Julia packages built with that particular version of Julia would be best used along with MSYS2-built versions of any other library that the Julia package wants to use.

There are other technical issues that I dislike about MSYS2 aside from the choice of exception handling model on win32, so we barely support using MSYS2 as a build environment at all. That could potentially change with appropriate PR’s and testing and contributions from people who feel differently than I do. We made an attempt to set up MSYS2-based buildbots to experiment with using them as an alternative for building Julia, but it did not work out well. MSYS2 is not well suited for automation and use by developers who aren’t on Windows themselves. You could (after a great deal of trial-and-error) use appveyor for small enough libraries that fit within the time limit, but remote debugging and iteration over a CI service is painful and not what I want to be the primary way of initially building and packaging binaries.

regarding CxxWrap: are you willing to accept the MSVC binaries

It’s your package, do whatever you want. Just be aware that MSVC issues, if you or your users hit any, are going to get a response consistent with my opinions here for the time being.

FYI, SymEngine.jl is using conda binaries for libsymengine which is a C++ library with manually written C wrappers and compiled using MSVC.

Ah good to know. If you’re very careful about what gets sent across your C API then it’s possible to avoid problems.