The big issue was upgrade deadlock. Instead of making it possible for packages to support both Python 2 and 3 at the same time, the Python devs provided the 2to3 tool, hoping that the entire ecosystem would upgrade from Python 2 to 3 all at once. But upgrade effort turned out not to be what stymied the transition. Packages still needed to support Python 2 because most of their users were still using it. And since most packages stayed on Python 2, users did as well – if anything they depended on was stuck on version 2, they basically had to. This created a deadlock where most packages and most users were stuck on version 2 indefinitely.
This deadlock wasn’t broken until relatively recent Python versions made it possible to support both 2 and 3 at the same time. Packages gradually started to do this, which eventually allowed users to upgrade once all the packages they depend on supported Python 3 and they did the maintenance work required to port their applications to the new version. Once most users upgraded – which is only now starting to happen – packages have finally been able to start to drop Python 2 support.
Worsening the upgrade deadlock situation for Python was the fact that so much of the ecosystem is written in C. The C API also broke compatibility but the 2to3 tool didn’t help with that. This situation seems to have been particularly painful in the SciPy ecosystem, which took unusually long to switch over. Somewhat ironically, it was probably easier to make C code support both versions of Python by using a messy but effective nest of #ifdef conditional compilation directives.
All of this is just my interpretation of the Python transition process as an outsider. I haven’t developed any Python packages and I wasn’t involved, so take it with a grain of salt and please forgive me for any inaccuracies.
The fundamental problem stated in generic terms is that packages can’t drop old version support until their users have upgraded to a new version, but users can’t upgrade until the packages they depend on support the new version. Since this process takes some time – and the larger the ecosystem, the longer it takes – this is a deadlock unless packages can support both the old and new versions during the transition. The main lesson for Julia (and other languages) is:
Always make it possible for packages to support both the previous and next versions of a language for long enough to let the ecosystem transition.
That’s what the Compat package allows and it works quite well. Having real macros helps a great deal. The only “hard breaks” we’ve had have been when we’ve introduced new syntax that was previously not parseable, which has only happened (IIRC) with the new type declaration syntax (type => mutable struct, immutable => struct, etc.) introduced in 0.6. To allow for that transition, we had to allow both old and new syntaxes in 0.6 and then deprecate the old syntax in 0.7, forcing packages to choose between supporting 0.5 and 0.7. (The old syntax will be an error in 1.0.) However, by the time 0.7/1.0 comes out, all actively developed packages (by definition almost) and the vast majority of users will already have upgraded to 0.6 so dropping support for 0.5 won’t be an issue. The main concern for the 1.0 transition is to support 0.6 and 0.7/1.0 at the same time for long enough for the ecosystem to catch up. I suspect the transition will be shorter than usual because of the promise that we’re done for the time being. [Also: FemtoCleaner]