I agree with the tone of the preceding conversation that we need an explicit definition of “public” which includes all code which absolutely cannot be broken in patches, and in principle I think basing this on export (for Base at least) is a good idea, though I’m a bit apprehensive about whether the current set of exported functions is sufficient.
I know I’m getting waaay ahead of things here, but I’ve given this some thought particularly as I have been quite involved with upgrading code for 0.6 to 1.0.
We really don’t want to be Python. At this point, it’s almost as if Python’s users are actively screwing its developers. I can’t tell you how many times I see something using Python2 and I say “Why is that on 2?” and they don’t even know. You can be sure much of that code (talking scripts now) is 3-incompatible when there was no real need whatsoever for it to be written that way. Most linux distributions, particularly Red Hat and Debian, still use Python2 as default. Worst of all, people who insist on using 2 occassionally have some justification for doing so as there seem to be a variety of (from what I can tell, poorly documented) performance regressions on 3. The package ecosystem is actually not too bad, most of the important stuff works fine on 3 (which is perhaps not saying much considering the amount of time 3 has been out), but regardless it seems the world is stuck with Python2 for all eternity. We should use this situation as a model of everything we want to avoid.
In my opinion at least 4 things are necessary not to wind up in that situation:
We are going to have to be more conservative about breaking changes than we were during pre-release. I love how much effort went into polishing everything for 1.0 so that we are not stuck with silly choices for all eternity, but that was the last pre-release transition and we will never have quite that much freedom ever again. Breaking interface changes probably should occur only as parts of comprehensive re-works, and not “well, we just thought that was nicer” (as much as I personally like to change things based on that justification alone).
Compat is going to have to be really good. Let’s be honest, in a lot of places we kind of half-assed Compat for 0.6 to 1.0. I’m not knocking all the great work that was done, in some areas it actually was really polished. I am personally extremely guilty of failing to help Compat be better: I did a huge amount of work updating packages but I barely ever made any commits to Compat. That’s because writing good Compat code is a really big pain, there are so many little gotchas. It was usually so much easier to just write some horrible hack into the package I was upgrading. That’s not going to be a good enough excuse for 2.0, it’s going to have to be really, really solid. I for one resolve to be much more helpful when that time comes.
Benchmarks. If there are regressions, people won’t use your upgrades for good reasons.
Really robust FemtoCleaner. Ideally, no human being should ever have to update code. I think we should aim to get as close to that ideal as possible. Considering the level of ambivalence-bordering-on-contempt I’ve seen in Python world, I really think this is the only thing that will truly save us.
Fortunately I think we are a large part of the way there. The basic components such as Compat and FemtoCleaner have been with us since well before 1.0. We just need to make sure that we really take this stuff very seriously for 2.0, or Julia 1.0 will be the only version of Julia that most people ever use.
I’ve moved this post to a new thread because it’s very tangential to the release process for 1.x and I really don’t want that very immediate and non-hypothetical discussion to get overwhelmed by a thoughts about the still-quite-far-off-in-the-future Julia 2.0 transition.
I really think people obsess way too much about the Python 2 to 3 disaster. Obviously, we can’t break things and slow down the language and not give people better language features… I’m not sure why the Python devs thought that would work. In the mean time Ruby had a very successful transition from 1.8 to 2.0, Java and C# have made many major releases, PHP has had a several of major releases, and so on. Julia itself has managed six major releases.
Here are some of my previous thoughts on the Python 2 to 3 transition:
It’s highly visible, it affects a lot of people and, as you stated, it’s a disaster. Still, you’re right, there are many counterexamples, I’m clearly to preoccupied with it.
I think the gist of my statement should have been more about this. Yes, you’re right about that, but I don’t think you should assume that the 1.0 to 2.0 transition will be very similar. Those releases occurred when it was well known that the language was in a development state. Many people who wouldn’t have touched it before 1.0 will have started using it before 2.0 is out. Again, it’s very nice that the foundations are already laid, but my fear is that people involved in the development will assume that 1.0 to 2.0 is going to be exactly like the previous transitions, while in reality I suspect that there is a good chance that some things which were happily tolerated during the alpha period will cause large swaths of users to get stuck on 1.0. I don’t think there’s anything fundamentally different that needs to be done, just that those of us who get involved with development need to take surviving massive breaking changes deadly seriously in the future.
I don’t think it’s that complicated. If I had been a Python user when Python 3 was released, I wouldn’t have switched either. It broke things, was substantially slower, and didn’t have any appealing features to make the pain of switching worthwhile—in short, it was all pain and no gain. You can’t just tell people to upgrade because they’re supposed to and expect them to do it. There has to be some motivation to make people want to upgrade, like improved performance or exciting new language features. Julia 2.0 will come with features that people are dying to have that aren’t possible without breaking things. And we care deeply about performance, so of course we won’t release a Julia 2.0 that’s slower than Julia 1.x—that would be terrible.
Just a side-note which is always in my mind when I see people comparing any major version jump with the Python 2/3 disaster…
I think the biggest issue in the Python 2/3 transition was the fact, that Python 2 came packaged with many Linux distributions and also with Mac OS, including dist tools which were basically part of the OS itself and the users installed them via the package managers.
I remember one of my friends once force-removed Python 2 from his RedHat since he thought upgrading to Python 3 and abandoning Python 2 would be a good idea. He was not able to boot the system correctly afterwards…
On the other hand, it was often not even possible to install Python 3 from official repositories until “recently” in so many distributions which are used in universities and other institutions (CentOS 6, Debian whatever etc.) all around the world. You know that those always take their time with upgrades… Our computing centre e.g. just upgraded to CentOS 7, halleluja.
Finally, dealing with multiple Python versions for not-so-experienced users (in my experience especially people coming with something like a Windows/MATLAB background) is really annoying. Things like conda and pyenv make it a bit better but even I (I consider myself more or less as a Python expert) struggle sometimes to debug a nastily messed up environment.
In my opinion, as long as Julia is not shipped with an OS distribution, there is not too much to worry
Btw. you find similar issues with C++ (or even Obj-C and Swift): on macOS you get the clang compiler, on Linux distros one specific GCC version which is bundled and things get complicated when you need anything older/newer.
Whatever, I like the discussions about smooth transition strategies, but seriously, I think some people are a bit too apprehensive
That shouldn’t have been a problem in and of itself. All sorts of packages get major updates all the time, and indeed much of that stuff has other packages which depend on it. (Regardless, we don’t have to worry about it since we are all switching to Arch now, right? )
Anyway, I think it’s true that I am overly preoccupied with the Python3 situation, and I probably over-analogized it to Julia. I’m probably traumatized by routinely witnessing people who supposedly love Python being openly hostile to its development, it’s a strange, messed up situation so of course any thought that that could ever happen to Julia just makes me sad.
a) python story is big success today! (we would like to see something similar with julia)
b) IMHO python2 forever is like saying cobol forever. Well. Maybe. Somewhere. With less and less significance…
c) Julia transition to 1.0 doesn’t seem to be success yet. And definitely doesn’t seems like doing this transition (much) better than python (although it was doing it 10 years later and we could learn from their mistakes!!!) Or another question - how would you like to measure success here?
I appreciate to speak about 1.x → 2.0 transition but we probably have to analyze 0.x → 1.0 first.
# julia 0.6
idx = search(str,'$',pos)
# julia 1.0
idx = something(findnext(isequal('$'), str, pos), 0)
I don’t want to be overcritical here (and yes this code could be written a little simpler) but maybe sometimes we have not just shouting how better we are, but also looking what wrong we did.
From wiki: Ruby 2.0 is intended to be fully backward compatible with Ruby 1.9.3. As of the official 2.0.0 release on February 24, 2013, there were only five known (minor) incompatibilities.[34]
So it was out a little more than 4 years later than python 3.0!
In ruby page you could read: Compatibility.
We have also taken care with the 2.0.0 design to make it compatible with 1.9. It will be easier to migrate from 1.9 to 2.0 than it was from 1.8 to 1.9. (The notable incompatibilities are described later.)
In fact, thanks to the dedicated work of third parties, some popular applications such as Rails and tDiary have been reported to work on the release candidate version of 2.0.0.
Had we working significant third party packages during 1.0 prereleases? No. Althoug Julia 1.0 was sometime predicted to be in September 2017 there was that quit hurry time to make it happen at Julia-con 2018 so there was not similar to Ruby’s transition. In contrary there are some packages like Keno’s (JuliaComputing’s co-founder and CTO - tools) Cxx.jl where we are not sure if it has a future.
We probably could to speak more about strings in 1.x and make work with them not so painful comparing to work in python?
This is because the mentioned commit applied deprecation warnings mechanically. That code doesn’t actually need 0 when there is no match, it would have been clearer using nothing instead. That’s the kind of thing that happens during transitions but that wouldn’t be used in new code.
What do you suggest we should have done in the 0.6 → 0.7 transition? Would you rather have kept the inconsistent mess that search and find* functions were in 0.6?
BTW, we could definitely allow findnext('$', str, pos) as a shorthand, but since it’s easy to add in any 1.x release, but impossible to remove until 2.0 (and even then it will be annoying for users), we have taken the safer approach of only implementing the minimal API.
Many people are using it in production (I am, and I find it nice).
Some packages did not make the transition to 1.0. Some are WIP, some seem to be abandoned for the time being. This is just highlighted, not necessarily caused by the 1.0 release.
Yes. It is why I wrote “this code could be written a little simpler”.
It is good to see these two sentences in bigger picture:
Mess is not good. But there is another danger. This mess (= " something(findnext(isequal('$'), str, pos), 0) " ) could stay forever.
In other words we are not really there (to have well polished 1. version). One has to be no surprised that there are people who will still wait before switch their work to Julia. So it is not success yet.
BTW. I am glad you are open to answer design question so I please forgive my curiosity in topic which probably is not about future designs. Why is Julia doing this:
julia> "a" in "Julia"
ERROR: use occursin(x, y) for string containment
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] in(::String, ::String) at ./strings/search.jl:459
[3] top-level scope at none:0
Somebody has to implement function which is doing this “useless” exception instead of using it like syntactic sugar (to attract more people). What is behind this decision?
That’s because strings act as collections of Char everywhere, and that in checks whether an element of a collection is equal to its argument. It doesn’t work for subsequences: [1, 2] in [1, 2, 3] doesn’t return true, because it would be ambiguous for arrays of arrays (e.g. [[1, 2], [1, 3]]). That’s the same distinction as push! vs. append!. I guess we could make an exception for strings, but that would make it harder to reason about the big picture.
