The Python 3 breaking changes were a big mess for them, because by that time people had built lots of code around Python 2. There seems to be a similar problem happening now with updates to TensorFlow.
What areas might lead to similar issues for Julia? What is being done, or can be done, to prevent or mitigate this?
The biggest potential issue I see with community scalability is our approach to “reserved names”. By discouraging type piracy, freely using packages, and needing to be careful not to use a name that’s used differently in an existing package (especially a popular one), I’d expect Julia programming to gradually become more and more constrained. Eventually something may need to shake this up, so it seems a possible pain point on the horizon.
Are there other things like this with some potential benefit to considering well ahead of any big problem? How can we avoid an eventual breaking-changes dumpster fire?
I find what Guido van Rossum said in a talk last year very interesting:
What we intentionally didn’t do but what I still regret is that we said there will be no way at run-time to combine Python 2 and Python 3 code in the same process, in the same Python interpreter. […] That sadly made it so that everyone had to really convert all their code and all their dependencies, like all the third-party packages they are using to Python 3 code before they would actually be able to run that code in Python 3 and benefits from the Python 3 features.
I think this is equivalent to what Rust is doing with the edition system and, for packages, what Go is doing with semantic import versioning. Maybe Julia and Julia packages can use similar approach.
Julia has at least two major advantages: macros, and fast FFI. The first provides a lot of room for tooling like Compat.jl (for better or worse). The latter means that there is very little module code using the libjulia C API, except for unavoidable cases like embedding. The fact that Python 3 introduced CPython C API breakage made supporting 2 + 3 code in-process much more difficult than a hypothetical situation where all C interfacing was done via something like ctypes (not very practical for performance and other reasons; having a JIT helps a lot).
I think there’s a third major and even more important advantage which makes Compat (and it’s evil cousin @deprecate) work much better than expected: we have a solution to the expression problem. Because of this one can often just write using Compat (or sometimes using Compat: newfunc, NewType) and then happily write module code which uses the new APIs on an old Julia version, without any further thought to compatibility.
This semantic compatibility for types and functions is even more complete than the syntax compatibility provided by @compat (which is limited to processing the syntax accepted by an older version of the Julia parser).
I would be surprised if Julia v2.0 is that different from Julia v1.0, at least in syntax and Base. The only major change I can see a huge demand for is the addition of traits, but this could be done without breaking existing code. The other change is “fixing” the global scope issues when code is run in REPL, that would impact scripts but not libraries.
StdLib is a different story, though I’d be surprised if it’s still included in Julia v2.0. It feels like we’re heading towards LinearAlgebra, Random, etc. each being separate packages.
I think that 2.0 will also be used as an opportunity to make all those tiny changes which are now shelved because they are considered breaking (API consistency issues, changes to semantics of functions, …; the label linked is just a subset of these).
Just like it happened before 1.0. There is no point in releasing 2.0 unless it is breaking stuff. Upward-compatible extensions can go in 1.x.
Right, I didn’t mean to imply they would not be breaking, just not on the same level as Python 2 → 3. Actually, I’d bet the changes are less extreme than Julia 0.6->0.7…
One thing Julia has in its favor, I think, is that Julia already has a, uh, “culture of source transformation”. It’s already relatively easy to achieve most desired semantic outcomes with macros and the like. I think it already provides a more obvious path for creating robust conversion tools to transition between versions.
Doesn’t cover everything, but it covers a lot, I think.
I would argue that macros can (theoretically) cover anything that parses. However, Julia 2 is likely to include some modifications to the parser itself.
Hence, it would be really neat if the Compat package would contain a copy of the entire Julia 2 parser and provide an include2 function and an @eval2_str string macro that could be called from Julia 1, parsing Julia 2 code and using further code transformation to make it run in Julia 1 (as long as it doesn’t depend on any new features.)
it’d also be neat if Julia 2 had a corresponding include1 function and @eval1_str macro.
Supporting calling older code from newer code makes sense. I don’t think it makes much sense to support calling newer versions from older versions. We want to encourage people to update, not make it easy for them to stay on the same version forever.
I would argue the opposite. If it is easy to call old code from new Julia, but not the other way around, then the path of least resistance for package maintainers will be to stay on Julia 1. Upgrading to Julia 2 would require maintaining two versions of the code, unless all end users have already upgraded. (For open source packages, the developers don’t know who the end users are, so as long as there’s a maintained LTS branch of Julia 1, they might assume that some people are still using it.)
if they don’t want their package to keep being used and or don’t want to use new Julia features that’s fine, and that’s exactly why it’s important for a user to be able to use these older packages because in extream cases, you simply have people abandon their package.
Imagine how much faster python 2->3 would be if people can just import python2 package temporarily but write their own code in python3.
Maybe I wasn’t clear. Of course it should be possible to use old packages. I’m not arguing against that. What I’m arguing for is to also have a way of upgrading a package without forcing all of its users to upgrade to Julia 2.
Being the first to upgrade should be as painless as possible. If a package maintaner updates his/her code on day one, only to start receiving requests like “Could you please backport your bugfixes to Julia 1, because Julia 2 exposed a bug in package X, which is currently unmaintained, so I can’t upgrade yet. Besides, once I make the request to install Julia 2, it’ll probably take six months until the IT department approves it.” then the package developer might think “OK. Even though I’d like to use the features of Julia 2, I don’t have the time to maintain two sets of code, so I’ll just stay on Julia 1 for now. At least that works for everyone.”
I understand that this is hypothetical, but the first thing I would try is fix that bug in package X. If it is unmaintained, then just fork.
I understand that some people are in this situation. But I am not sure that shifting the maintenance burden of a broken policy inside some company/organization to open source developers who are mostly volunteers is the right way to go.
I think that the best equilibrium is when users understand that they should have a very strong preference to migrate to latest versions ASAP. This may be delayed for a compelling reason, but then they should bear the cost of this: pay someone to maintain backports branches, or do it themselves.
That is definitely the best thing to do, and I’m glad that you would do it. But in my experience, people frequently tend to prefer less work over the right thing. (Usually not out of laziness, but because they have deadlines to meet.)
If you work for a university, then making a PR to an open-source project might only take ten minutes, but with some other employers you would first have to fill out a form explaining why you want to release code (that you wrote but your employer owns) to an open-source project. Then you wait a few months for the legal department to approve your request before you can make the PR. (This is not hypothetical. I know of a company that uses Julia and has a policy like this.)
Of course it would be much better to fix these broken corporate policies than to spend valuable developer time on building work-arounds. But how would you do that? Are there people in the Julia community that have enough influence on the management of large companies to make them consider updating their internal policies?
This was how the switch to 1.0 was done (and it went remarkably smooth.) But since then, the Julia developers have explicitly stated that if you only want bugfixes and don’t need new features, then it’s perfectly fine to stay on an LTS release, and therefore many of the popular packages currently include the 1.0.x LTS release in their unit testing.
Personally, as a maintainer of various packages, I would just decline the opportunity to work for free because some company decided to complicate their own life.
Frankly, I am not sure anything needs to be done here. This would pretty much happen automatically, and the community should just communicate clearly that expecting projects to backport stuff for an extended period of time is not reasonable.
Then I am just really sorry for them, and expect them to suffer the consequences, and eventually be driven out of business by competitors who are more flexible. Yes, dumb companies do exist, but that’s not a sufficient reason to design open source projects around them.
Also, I imagine think these discussions would be more fruitful if a representative of a company like you describe spoke up here, instead of others like you arguing their position for them. Once they are aware of their needs and constraints, it is possible that a constructive solution could be found, or at least they could go to their management and ask for more flexibility.
