That would make me happy!
1 Like
Thanks @jlperla. For what it is worth, I sit in the intersection of Group 1 & 2. However, the concerns you highlight from Group 1 are not really a concern for me. If I spend months building a package and the API changes, it is usually not a massive effort to update to the new API and with Compat, a lot of the work is done for me already. It might take me a few days. No big deal.
I do sympathize wholeheartedly with the concerns you highlight about Group 2. I am a fintech startup founder with a staff of 24 and growing. I’m about to introduce my package (that’s been in the making since v0.3) extolling the virtues of Julia to my team and it takes 60 seconds to load after using 
:
PS: Just before hitting “Reply”, I saw @StefanKarpinski’s reply. Sounds awesome 
It’s good to be back!
1 Like
It’ll be ready when it’s ready – sooner is better but it is not unclear to us that we need to “stick the landing” which entails a combination of language readiness, tooling quality, documentation and messaging. Separating 0.7 from 1.0 a bit is a possibility if the language is ready but the rest isn’t quite, but we’ll see.
12 Likes
I think there may be some general confusion about the way that things will be released. The various release steps for 0.7 and 1.0 are not necessarily in lock step. They do have a joint feature freeze because we don’t plan to add any features between the two. However, as we have learned from past releases the ecosystem takes a couple of weeks-months to catch up once feature freeze has been called. This time around will be no different. The time between feature freeze and final release precisely exists to make sure the ecosystem catches up to the language, bugs that have accumulated get fixed, documentation gets updated, etc. Nobody is suggesting releasing 1.0 next week without taking the time to fix or with the ecosystem in disarray.
In fact, once feature freeze is called, the core developers will have significantly more time to focus on some of the common complaints that have been put on a back burner in order to make time to work on the necessary breaking features to go into 1.0.
Let’s take code load time/latency for example. There’s been some very good work already for 0.7 (interpreter improvements, linear IR, etc.) that will make low-latency julia after 1.0 much easier. For an early preview of this, grab a recent version of 0.7 and run julia --compile=no. This doesn’t solve package load times yet, but try something like sort(randn(10)) and compare the REPL experience with the current default. There’s lots of good stuff that we have a fairly clear path towards. However, none of that can be done before feature freeze.
Another example is the debugger. ASTInterpreter2 now works ok on 0.6+, but making things work really well is a bunch of work, and getting the language stable is really of paramount importance in order to a) give the community a stable base to work on, rather than playing catch-up all the time and b) free up core developers to work on some of these things. As a result, I’ve spent my time these past few months working on various Base things (+ FemtoCleaner, etc), rather than the debugger. I have a pretty good idea how I want to do this and what is required, but getting 1.0 to feature freeze has taken priority.
That said, I think everybody is aware that 1.0 presents an opportunity to have new people try this out and we need to make sure they have a good experience. We do need to release 1.0, but we won’t be rushing it out the door if the ecosystem isn’t ready. We’ll have to see how it goes. That’s why I intentionally didn’t specify any dates for release milestones beyond the feature freeze in my original announcement. We’ll do what’s necessary here. If a few extra weeks are required to get the ecosystem in order so be it 
.
25 Likes
Spectacular. That is all we needed to hear. The other side of the coin is that the people who want a better experience prior to an official 1.0 don’t want it to hold up the official 0.7 release of the language.
In the meantime, where is the best place to post issues/etc with the default juliapro installation and tutorials for the user experience? I can tell my students, RAs, and coauthors to submit issues as required.
I’m not sure of a repo (and just to clarify, I am not a member of JuliaComputing), but the channel #juliapro on the Slack is a quite responsive way to report issues with the distribution.
Thanks for chiming in and confirming!
I think that previous alpha phases went a little quickly, and I’d point to this issue as one that dragged on through the ecosystem for a few months after the v0.6 release (though I understand the pressure to release at JuliaCon!):
@jlperla makes some good points that it would be good to have things stabilized before the full v1.0 release, but at the same time that should be balanced with the fact that if we want perfection it will never release. I think that pre-planning a long alpha would be nice, especially if there was some (flexible) length given. If I had to throw a time out there, I think that 2-3 months post release is how long it took before v0.5 and v0.6 really felt “good” (i.e. depwarns were cleared up, almost every package working, etc.). Having that as part of an alpha/beta phase may be useful. The biggest issue I have though is that most of the bugs from updating versions weren’t found until after the Julia version release because we just didn’t have enough users actually using it. It would be nice to have a “please, users start testing packages” phase added to what we previously had.
Also, something to keep in mind with JuliaPro is that its package versions are fixed. If it stays that way in v0.7/v1.0, we really need to make sure it’s set to versions which are updated. With the Julia v0.6 release, since it was released at around the time of the v0.6 itself when the ecosystem had the problem, JuliaPro had upper bounds that kept quite a lot of packages back at versions which hit the #21969 issue, and this persisted well into September. Maybe JuliaPro should have an alpha release as well to catch this kind of issue.
This became somewhat of a (hopefully informative) ramble. I have no idea what I am suggesting, but there are some pain points which we should address better. Anyways, happy holidays and I’m sure in the end we’ll get to the right place.
4 Likes
First of all: thank you (and to all of the people working on Julia). I hope you don’t take people’s frank statements about the state of the development experience personally, as everyone understands that language stability is required before a debugger. Our worry was simply that you wouldn’t be given enough time to do it prior to a v1.0 release, and that worry has been eliminated.
For what it is worth (and from someone who really likes inspecting code in the debugger), I think any time spent getting the debugger and IDE experience better on v0.6 is time-wasted. We have lived this long without a debugger, we can make it until 1.0.
I have seen a lot of complaints about the amount of time spent loading code, how Pkg.update() is a deal-breaker, etc. My general feeling is that many of them are missing the point. The code has to be compiled, and the fancier the metaprgramming, the fancier the more time it will take.
The question is: (1) when; and (2) what visual feedback do users get when they are doing Pkg.update() and using... Those are psychological issues, or questions of when to precompile, that can be handled in the alpha and beta phase of the release. My general feeling is: you can take as much time as you want with a Pkg.update() and JuliaPro installation as long as you spew out a lot of nonsensical filenames to let people know it is working. If you just precompile as much as possible then, so that using is fast afterwards, then people can walk away from their computers for 15 minutes each time. Matlab takes a good half-hour to install, and stata takes a huge amount of time each time it updates. But people are visually/mentally prepared for both.
But mostly: thanks again to you and everyone else for doing such hard and generally thankless work, with a bunch of free-riders like me who mostly seem like ingrates. Happy holidays.
14 Likes
Not sure if you’ve tried out Pkg3 yet, but it is significantly faster, so Pkg.update() generally takes no time at all ;).
2 Likes
Alas, I am more of the “wait for the beta” type, and can’t justify time spent tinkering before then. Pkg3 sure looks promising. I flipped through the source and output, and have a few concrete suggestions based on my Pkg2 experience that I can tell haven’t changed. Is that best posted in a new discourse thread, or in an issue in Pkg3 repository?
The Pkg3 repository would be best.
I posted https://github.com/JuliaLang/Pkg3.jl/issues/89 and https://github.com/JuliaLang/Pkg3.jl/issues/90 for free-disposal.
+100
it wouldn’t be catastrophic if there were a 0.8 or even 0.9. julia would be better for it i would even say.
2 Likes
Wise human learns from others.
Experience with python3 adoption path could warn us. It really take few days to port some packages and >10 years to port whole ecosystem.
But they underestimate python’s popularity which is probably not our problem now…
AFAICT Python 3 was a major rehaul of the whole language. If you have a specific Julia language feature in mind that you imagine could cause similar problems, please explain what you mean.
Otherwise, I see no reason for alarmist extrapolation. Even major changes, like the “arraypocalypse”, did not cause hiccups anywhere near that timeframe.
5 Likes
I don’t see (and python core developers didn’t see too) any big feature changes which caused >10y delay.
Everything could be rewritten. It is not language but industry problem:
- it cost money
- it cost time (see 1)
- it cost human sources (see 1 and 2)
- it could create new bugs (see 1-3)
- it could revive old bugs
- you have to wait for new buggy dependent library (see 1-5)
- packages which are dependent on your packages have to wait for your new buggy versions (do you see? )
- some packages have not maintainers (or people who understand are too busy with other problems) or have new maintainers who are not experts
- edit (I forgot probably most important): old solution works for many (= it cost them no money)
One big difference from Julia 1.0 is that python3 was slower and didn’t bring any killer property! 
Second one is that Julia’s ecosystem is significantly smaller. 
But I am not talking about not to go to 1.0!!!
I just concentrated focus on one possible oversimplification that if one package could be ported in a few days then there is not big deal.
There is a nonlinear effect that dependencies have to be updated first. So as there more packages you have under your umbrella, this window grows faster than linearly. In JuliaDiffEq it doesn’t take days to port, I really wish it would. It could be a month long process or so to actually get things right. You’ll notice that after each update it’s “quite easy” to find new bugs for about 2 months or so, and this is because updating takes a lot of time and effort. It’s not anywhere close to Python’s update problems, but it’s a very real amount of man hours.
Looking at the current state of Julia, I’m not sure how worth it it is anymore but it seems like most of the “must have” next features (debuggers, compiler optimizations, precompilation overhaul, etc.) are not language breaking, so that’s why 1.0 is a great idea: it’s a signal that we can stop working on updating code to new Julia versions every few months and start getting these cool new features which don’t require updates!
4 Likes
I somehow agree, however looking back at the journey from 0.3 to 0.7 which i monitored, language features and unfortunately syntax changes were (imho) triggered by development on these usability/ecosystem topics (like debugger etc.); so let’s see.
The big issue was upgrade deadlock. Instead of making it possible for packages to support both Python 2 and 3 at the same time, the Python devs provided the 2to3 tool, hoping that the entire ecosystem would upgrade from Python 2 to 3 all at once. But upgrade effort turned out not to be what stymied the transition. Packages still needed to support Python 2 because most of their users were still using it. And since most packages stayed on Python 2, users did as well – if anything they depended on was stuck on version 2, they basically had to. This created a deadlock where most packages and most users were stuck on version 2 indefinitely.
This deadlock wasn’t broken until relatively recent Python versions made it possible to support both 2 and 3 at the same time. Packages gradually started to do this, which eventually allowed users to upgrade once all the packages they depend on supported Python 3 and they did the maintenance work required to port their applications to the new version. Once most users upgraded – which is only now starting to happen – packages have finally been able to start to drop Python 2 support.
Worsening the upgrade deadlock situation for Python was the fact that so much of the ecosystem is written in C. The C API also broke compatibility but the 2to3 tool didn’t help with that. This situation seems to have been particularly painful in the SciPy ecosystem, which took unusually long to switch over. Somewhat ironically, it was probably easier to make C code support both versions of Python by using a messy but effective nest of #ifdef conditional compilation directives.
All of this is just my interpretation of the Python transition process as an outsider. I haven’t developed any Python packages and I wasn’t involved, so take it with a grain of salt and please forgive me for any inaccuracies.
The fundamental problem stated in generic terms is that packages can’t drop old version support until their users have upgraded to a new version, but users can’t upgrade until the packages they depend on support the new version. Since this process takes some time – and the larger the ecosystem, the longer it takes – this is a deadlock unless packages can support both the old and new versions during the transition. The main lesson for Julia (and other languages) is:
Always make it possible for packages to support both the previous and next versions of a language for long enough to let the ecosystem transition.
That’s what the Compat package allows and it works quite well. Having real macros helps a great deal. The only “hard breaks” we’ve had have been when we’ve introduced new syntax that was previously not parseable, which has only happened (IIRC) with the new type declaration syntax (type => mutable struct, immutable => struct, etc.) introduced in 0.6. To allow for that transition, we had to allow both old and new syntaxes in 0.6 and then deprecate the old syntax in 0.7, forcing packages to choose between supporting 0.5 and 0.7. (The old syntax will be an error in 1.0.) However, by the time 0.7/1.0 comes out, all actively developed packages (by definition almost) and the vast majority of users will already have upgraded to 0.6 so dropping support for 0.5 won’t be an issue. The main concern for the 1.0 transition is to support 0.6 and 0.7/1.0 at the same time for long enough for the ecosystem to catch up. I suspect the transition will be shorter than usual because of the promise that we’re done for the time being. [Also: FemtoCleaner]
27 Likes
I have barely started to look into Julia: I have liked what I see in that Julia already has packages that are stronger than what I find in Python and MATLAB. Still, I have delayed really digging into the language since it has been in transition.
To me as a newcomer, it is an advantage if v. 0.7/1.0 is frozen and released as soon as possible – I can have patience with a reasonable period of bug fixing as long as I can start to play with it knowing that the key syntax is fixed. A quick release of v. 0.7/1.0 will also help authors of Intro to Julia books to finish their books, and help explain how to use the language (several publishers seem to have put publications on hold until v. 0.7/1.0 is out). Good intro books are key to the success of a language!
A couple of questions/observations:
-
I’ve been using Python 2.7 with Jupyter for a couple of years, and have played around with IJulia – a notebook tool is useful when learning a language (but doesn’t really replace an editor/debugger).
-
Jupyter Lab is “imminent” (latest release date of beta version is January 5, 2018?) – so I’m curious if there are any plans to update IJulia to IJulia Lab…? Perhaps editor/debugger could be a plug-in in Jupyter Lab – it seems like the Jupyter Lab people think along such lines.
-
In Python, there is a function which responds with all reserved keywords in Python – useful when learning a new language. Is there a similar function in Julia?
-
As a long term MATLAB user, I appreciate that Julia has copied things from MATLAB. But I appreciate the possibility to also improve on MATLAB, as indicated by the discussion on whether function expm() can be renamed to exp() since MATLAB’s function exp() effectively is exp.(x) in Julia… In a similar vain:
– I assume that the Control package of Julia has copied MATLAB’s LTI system. But why not improve on it? MATLAB’s LTI model essentially assumes finite dimensional state space model with allowance for time delays. But time delays are really just representations of scalar advection PDEs. Why not generalize the model to include linear DAEs and PDEs?
– Is it possible to combine the DifferentialEquation package with AD to automatically extract such a generalized LTI model (DAE, PDE, PDAE, etc.) at a specified operating solution, and combine this with the Control package? How should that be done?
Anyway, I appreciate the hard work you guys do towards releasing v. 0.7, and look forward to start using Julia more actively.
5 Likes