I have noticed that testing on OS X takes longer for packages on Travis (presumably machines running it are a scarce resource), so I am considering disabling it and testing on linux only.
I am assuming that this should be innocuous for a pure Julia library, since differences in test results would be bugs in Julia or the other libraries I use. Is this correct?
Out of curiosity, in light of the issue above, why does PkgDev.generate enable it in the first place?
(Sorry if this is a trivial question, the Julia ecosystem is my first encounter with CI).
2 Likes
In my experience I only found once differences between 32 and 64-bit systems, but thanks to AppVeyor, not Travis that seems to run only on 64-bit machines. I don’t recall ever noticing failures only OS X and not on GNU/Linux and vice versa for Julia-only packages that don’t play around with something system-related.
Doing system testing on all (available) platforms is in general not a wrong idea, because there is a high propability that your package is connected to others via REQUIRE and they might be OS-sensitive. And even if your package is julia only and doen’t have other requirements, julia nightly (so unstable version) could act differently on different OS. But you might also be lucky.
“In theory, theory and practice are the same. In practice, they are not.”
3 Likes
Ping @dpsanders - related to your test slowdowns?
I agree that more testing cannot hurt, but all things come at a price: delay in test results (unless one goes to the details), and use of apparently scarce resources that are kindly provided to us for free. If other packages rely on OS X specific things that should be tested, IMO they should test for it. If I don’t use the Macs, this means that those who really need them get their results faster.
1 Like
I agree, to minimize OSX on travis you can e.g. run all julia versions on linux and only nightly on OSX.