Avoiding a Julia version of the Python 2/3 mess

I find what Guido van Rossum said in a talk last year very interesting:

What we intentionally didn’t do but what I still regret is that we said there will be no way at run-time to combine Python 2 and Python 3 code in the same process, in the same Python interpreter. […] That sadly made it so that everyone had to really convert all their code and all their dependencies, like all the third-party packages they are using to Python 3 code before they would actually be able to run that code in Python 3 and benefits from the Python 3 features.

I think this is equivalent to what Rust is doing with the edition system and, for packages, what Go is doing with semantic import versioning. Maybe Julia and Julia packages can use similar approach.

3 Likes

Julia has at least two major advantages: macros, and fast FFI. The first provides a lot of room for tooling like Compat.jl (for better or worse). The latter means that there is very little module code using the libjulia C API, except for unavoidable cases like embedding. The fact that Python 3 introduced CPython C API breakage made supporting 2 + 3 code in-process much more difficult than a hypothetical situation where all C interfacing was done via something like ctypes (not very practical for performance and other reasons; having a JIT helps a lot).

6 Likes

I think there’s a third major and even more important advantage which makes Compat (and it’s evil cousin @deprecate) work much better than expected: we have a solution to the expression problem. Because of this one can often just write using Compat (or sometimes using Compat: newfunc, NewType) and then happily write module code which uses the new APIs on an old Julia version, without any further thought to compatibility.

This semantic compatibility for types and functions is even more complete than the syntax compatibility provided by @compat (which is limited to processing the syntax accepted by an older version of the Julia parser).

2 Likes

I would be surprised if Julia v2.0 is that different from Julia v1.0, at least in syntax and Base. The only major change I can see a huge demand for is the addition of traits, but this could be done without breaking existing code. The other change is “fixing” the global scope issues when code is run in REPL, that would impact scripts but not libraries.

StdLib is a different story, though I’d be surprised if it’s still included in Julia v2.0. It feels like we’re heading towards LinearAlgebra, Random, etc. each being separate packages.

1 Like

I think that 2.0 will also be used as an opportunity to make all those tiny changes which are now shelved because they are considered breaking (API consistency issues, changes to semantics of functions, …; the label linked is just a subset of these).

Just like it happened before 1.0. There is no point in releasing 2.0 unless it is breaking stuff. Upward-compatible extensions can go in 1.x.

8 Likes

Right, I didn’t mean to imply they would not be breaking, just not on the same level as Python 2 -> 3. Actually, I’d bet the changes are less extreme than Julia 0.6->0.7…

3 Likes

6 posts were split to a new topic: Making views the default indexing behavior in 2.0?

One thing Julia has in its favor, I think, is that Julia already has a, uh, “culture of source transformation”. It’s already relatively easy to achieve most desired semantic outcomes with macros and the like. I think it already provides a more obvious path for creating robust conversion tools to transition between versions.

Doesn’t cover everything, but it covers a lot, I think.

3 Likes

I would argue that macros can (theoretically) cover anything that parses. However, Julia 2 is likely to include some modifications to the parser itself.

Hence, it would be really neat if the Compat package would contain a copy of the entire Julia 2 parser and provide an include2 function and an @eval2_str string macro that could be called from Julia 1, parsing Julia 2 code and using further code transformation to make it run in Julia 1 (as long as it doesn’t depend on any new features.)

it’d also be neat if Julia 2 had a corresponding include1 function and @eval1_str macro.

Supporting calling older code from newer code makes sense. I don’t think it makes much sense to support calling newer versions from older versions. We want to encourage people to update, not make it easy for them to stay on the same version forever.

14 Likes

I would argue the opposite. If it is easy to call old code from new Julia, but not the other way around, then the path of least resistance for package maintainers will be to stay on Julia 1. Upgrading to Julia 2 would require maintaining two versions of the code, unless all end users have already upgraded. (For open source packages, the developers don’t know who the end users are, so as long as there’s a maintained LTS branch of Julia 1, they might assume that some people are still using it.)

cough import future

if they don’t want their package to keep being used and or don’t want to use new Julia features that’s fine, and that’s exactly why it’s important for a user to be able to use these older packages because in extream cases, you simply have people abandon their package.

Imagine how much faster python 2->3 would be if people can just import python2 package temporarily but write their own code in python3.

2 Likes

Maybe I wasn’t clear. Of course it should be possible to use old packages. I’m not arguing against that. What I’m arguing for is to also have a way of upgrading a package without forcing all of its users to upgrade to Julia 2.

Being the first to upgrade should be as painless as possible. If a package maintaner updates his/her code on day one, only to start receiving requests like “Could you please backport your bugfixes to Julia 1, because Julia 2 exposed a bug in package X, which is currently unmaintained, so I can’t upgrade yet. Besides, once I make the request to install Julia 2, it’ll probably take six months until the IT department approves it.” then the package developer might think “OK. Even though I’d like to use the features of Julia 2, I don’t have the time to maintain two sets of code, so I’ll just stay on Julia 1 for now. At least that works for everyone.”

I understand that this is hypothetical, but the first thing I would try is fix that bug in package X. If it is unmaintained, then just fork.

I understand that some people are in this situation. But I am not sure that shifting the maintenance burden of a broken policy inside some company/organization to open source developers who are mostly volunteers is the right way to go.

I think that the best equilibrium is when users understand that they should have a very strong preference to migrate to latest versions ASAP. This may be delayed for a compelling reason, but then they should bear the cost of this: pay someone to maintain backports branches, or do it themselves.

5 Likes

That is definitely the best thing to do, and I’m glad that you would do it. But in my experience, people frequently tend to prefer less work over the right thing. (Usually not out of laziness, but because they have deadlines to meet.)

If you work for a university, then making a PR to an open-source project might only take ten minutes, but with some other employers you would first have to fill out a form explaining why you want to release code (that you wrote but your employer owns) to an open-source project. Then you wait a few months for the legal department to approve your request before you can make the PR. (This is not hypothetical. I know of a company that uses Julia and has a policy like this.)

Of course it would be much better to fix these broken corporate policies than to spend valuable developer time on building work-arounds. But how would you do that? Are there people in the Julia community that have enough influence on the management of large companies to make them consider updating their internal policies?

This was how the switch to 1.0 was done (and it went remarkably smooth.) But since then, the Julia developers have explicitly stated that if you only want bugfixes and don’t need new features, then it’s perfectly fine to stay on an LTS release, and therefore many of the popular packages currently include the 1.0.x LTS release in their unit testing.

Personally, as a maintainer of various packages, I would just decline the opportunity to work for free because some company decided to complicate their own life.

Frankly, I am not sure anything needs to be done here. This would pretty much happen automatically, and the community should just communicate clearly that expecting projects to backport stuff for an extended period of time is not reasonable.

Then I am just really sorry for them, and expect them to suffer the consequences, and eventually be driven out of business by competitors who are more flexible. Yes, dumb companies do exist, but that’s not a sufficient reason to design open source projects around them.

Also, I imagine think these discussions would be more fruitful if a representative of a company like you describe spoke up here, instead of others like you arguing their position for them. Once they are aware of their needs and constraints, it is possible that a constructive solution could be found, or at least they could go to their management and ask for more flexibility.

3 Likes

Yeah, me too. To be clear, this is not a “dumb” company. It’s very successful, with a market cap in the hundreds of billions and full of smart people. But a very good Julia programmer that I know just left the company for a smaller one precisely because of those stupid policies.

As for the upgrade to Julia 2, maybe it’s not necessary to ponder too much on hypothetical scenarios. Making predictions is hard. Especially about the future. Once Julia 2 has been released, we’ll find out if packages are upgraded or not, and if they are not, then we will be able to find out why just by asking the maintainers directly.

1 Like

Unfortunate, but a good solution if the values of your employer does not align with yours.

1 Like

Old versions of packages don’t cease to exist just because new versions are released for Julia 2. Users who are conservative and don’t want to upgrade can keep using older versions of packages; there can even continue to be bug fix releases, and if a company depends on an older version, it may behoove them to help make those releases. New features, however, should get added to the latest Julia 2 version of a package. This encourages users to upgrade if they need new features or packages that only exist for Julia 2 which creates a strong incentive for users to switch.

If it’s easy to call new packages from Julia 1, then there’s zero incentive for users—who are generally less proactive than package maintainers—to upgrade anything: they can just leave their applications running Julia 1 and use new packages as needed. By making it easy to call new Julia code from an application running Julia 1, you’d be making twice as much work for package maintainers since they’d end up needing to support two entirely different runtimes for an extended period of time (since there’s no reason for users to upgrade, so they won’t). Not to mention much more work for Julia devs who have to support not one but two bidirectional runtime systems. I can tell you that even just making Compat.jl work during the 0.x days was not fun.

10 Likes