The State of the Julia Ecosystem

Earlier this week I gave a presentation to London Julia on my experiences while revising my book on Julia to accommodate version 1.0. As I researched for the talk, it became clear to me that the presentation would not receive widespread approbation, speaking as I was to a set of believers. However it seems incumbent upon on to me to share my conclusions about the state of affairs in Julia with this wider forum, not with any expectation of achieving any great acclamation.

My sortie with Julia v1.0 began on the first day of JuliaCon when I flagged up my dismay of a philosophy seeking to impose a restrictive programming paradigm on a set of the very people we are hoping to attract. This has come to be termed the “Julia Global Debacle” and it would seem that the matter will eventually be addressed, although, in the opinion of this correspondent, while it remains in place the damage persists.

The bulk of this article is devoted to the current state of the so-called Ecosystem, after the ill-judged launch of version 1.0 in August. Indeed after the conference, Steven G Johnson suggested that there is little to be gained in using v1.0 and much to be lost, viz. the deprecation warnings provided by v0.7. Further Stefan Karpinski commented that it might have been a good idea to have launched v1.0 WITH deprecation warnings, although in fact isn’t that what v0.7 was and is?, suffixing his comment with those terrible few words in the English language: “what’s done is done”.

In August/September various questions were raised as to which packages work with v1.0; these requests have become less frequent of date, not because the subject has gone away but I suppose because the proponents are running out of stamina. Just after the conference, Tim Holy asked us to wait a couple of weeks while the current packages were brought up to compliance, an intended function of the previous version (v0.7) which it had failed to fulfil, as it only exited the development phase a couple of days before v1.0 was launched.

Stefan Karpinski stated then that this was in fact difficult to do, which is odd for a community capable of producing a system as complex as Julia and who had been providing such information on previous versions for a number of years. He also stated that the package NewPkgEval.jl does INDEED work, almost, and with a little badgering on his part, could be polished and run a regular basis. That was in August and now four months later, either it actually does not work or is just not being run, certainly the package’s source have not been touched for over two months.

So what help does Joe/Josephine Programmer get when they take their first tentative steps after downloading Julia? The old web page, pkg.julialang.org is now defunct but remains in existence, visible on the Internet and still referring to life pre-JuliaCon when Julia v0.6.3 was Queen and v0.7 was undergoing development.

The Julialang.org Home Page references Dan Segal’s, laudable Ruby on Rails package, running on JuliaObserver.com and which itself has not touched significantly since JuliaCon. Being an automatic process, harvesting data from a variety of online sources, it deals with activity not compliance and creates such interesting quirks as classifying QuantEcon.jl as the leading package in the Chemistry category. Of course JuliaObserver does not provide information about the extent to which packages actually work with v1.0, and with earlier versions - so failing to meet the queries posed right back in August - but is the primary source offered.

Since establishing the Julia User group in London, five years ago to the day, I have devoted my efforts to Paulian evangelistic presentations in the UK and elsewhere, speaking to Data Scientists, Pythons, R-ites and the like, which in general have been received with a deal of interest. At present this will have to cease as the sentiments I have expressed here would almost certainly be reflected in the options of these larger cauci.

I remain a supporter of the Julia project, albeit a less optimistic one than I was four months ago.

Malcolm Sherrington
London, UK, 14th December 2018

4 Likes

I am running a julia implemented algorithm which is a central part of an analysis platform, nearly every day, which saves hours of calculation time, which doubled or tripled our data throughput. Developing time was ridiculous compared to developing it e.g. in C.

It is running stable for years. Unchanged, no bugs, in all the time. No need to update it to newer julia version, because it is just working.

It saved us real money because there was no need for faster hardware.

We got this for free.

Every single step during my start with julia up to now, made julia better. There are great discussions where everybody can join, where I have learned so much and still learning.

Everything for free.

These are “my” facts I have experienced directly in practice and in real world computational problems.

Thats why I am extremely optimistic for the future of julia and your text above made me urgently write this up to shout a BIG THANK YOU to all the great people who made this possible and still are doing!

Good constructive criticism is extremely valuable.
What you say is a debacle is probably just the most controversial design decision and nothing else. There are good arguments on all sides and many good but different proposals were made. I think people are becoming religious about Julia and loosing the proper scope when things aren’t going their fixed way. This is probably the normal downside of success.

14 Likes

No, the real answer is because things got updated. Yes, it takes a few months for the whole package ecosystem to update. I am surprised anyone is surprised about that. But nowadays it’s hard to find a package which isn’t working on v1.0, so it’s just not a topic that comes up anymore. It’s actually surprising how fast Julia has successfully updated programming language versions throughout the community (I’m looking at you Python 3).

21 Likes

Julia 1.0 is a good release in my opinion. There were some issues which were solved in the 1.0.1 release, preventing me from making some of my code work earlier. However, I have faith that issues with Julia will be continuously ironed out, due to the community around it. Having 0.7 to fall back on with warnings is optional, however I don’t recommend holding on to those deprecation warnings, because the language moved on.

Presumably much of that chatter was by people who also develop and update packages, so they were probably helping each other out with making progress on it.

2 Likes

I’m too grateful for the work that efforts of the Julia group, which is why my post was directed to what needed to be addressed. If software can be beautiful then I consider Base, Stdlib and Core are all to be such.

No one assumes that upgrading the packages to v1.0 would be a simple matter, indeed that is why the interim v0.7 was muted but then not given time to fulfil its role. Indeed the flagship JuliaDB took over 3 months to be deemed as working after receiving the attention of the inner sanctum of the Julia community.

Readers of this blog are comfortable in their skins and my comments were aimed accommodating those who are not.

With regard to Python 3, there was, and is, a credible alternative distributed in Python 2, with its own separate ring-fenced repository, this is not the case with Julia. Packages may work with v1.0, others only v0.6 and (possibly) earlier, a few with both. But in the light of little guidance to the contrary, the only resort one has is to suck it and see.

Julia Computing provided an Anaconda style product in Julia-Pro, which whatever its demerits may have been, did at least allow new users to hit the ground running. This has been taken away with the advent of version 1.0.

1 Like

I am not sure what the purpose of this post is. It addresses 3 somewhat loosely related issues:

  1. communication about the upgrade path to 1.0,

  2. I assume that “Julia Global Debacle” refers to what some call the “global scope debacle”, ie #28789,

  3. Julia Observer.

Since 1.0 was released a while ago, talking about (1) is mostly useful to do it better next time. It is unclear what you are proposing.

Regarding (2), new solutions are being discussed, so if you want to contribute those topics (or yet another coherent proposal) would be the best place.

For (3), Julia Observer is also being actively discussed now, so you may want to suggest improvements there, or discuss how you could contribute if you want to.

On this forum, many people usually get fast and helpful answers to their questions (including, but not limited to, advice about packages). I think this is a great help for people learning the language.

12 Likes

Can you please clarify what you think has been “taken away”? JuliaPro, including older versions, is of course available.

1 Like

I did not wish my comments to be a discussion on Julia Pro, which is why I omitted them from the original positing. However JPro 0.6.x came bundled with a set of packages which allowed the user to install and start using the REPL, Jupytper or Juno immediately. Now (s)he is faced with the same procedure of selecting and installing packages first.

As for your previous comments, the purpose of the article is to highlight the lack of information that the new/causal user now gets in using Julia.

I did acknowledge that the Global Scope Debacle may receive some attention after Iain’s comments, my initial posting just received the reply that it was as intended, although not I should add not from Jeff, who defended his position admirably. It is interesting to see how the debate over this change in the month before it was deemed as to hot to continue.

My comments on Julia Observer, and I applaud Dan Segal’s work, is that by its very nature it provides information about trending not compilance which is what previous correspondents had requested, yet it remains the main reference to package usage from the home page of the Julia’s website

Sorry, I still don’t understand. As I said, earlier versions of JuliaPro are still available, were they updated to remove bundled packages?

Try downloading J-Pro 1.0.x and installing it, then start the package manager and issue an st(atus) command.

As I said I did not want to get detracted on the merits, or otherwise, of J-Pro; although for my talks to the non-Julia community it was a convenient suggestion as to how they could start.

I believe that J-Pro keeps a separate repository, but when adding a package which is not there defers to the main one, but that is heresay on my part. If so it does mean that one is never quite sure where the source is coming from!

Apparently we still don’t understand each other. I am curious what has been “taken away” from the users of 0.6.x when 1.0 was released.

Sorry, but I don’t understand who/what is detracting you. I don’t think I said anything to that effect.

This conversation on J-Pro, which was not part of my original comments as it would detract from the rest of the article.

I have been told in a private communication that v0.6 is dead and should not be used, so downloading older versions of J-Pro (v0.6) seem to have little merit.

I’m just confused on that comment though. JuliaPro v1.0 still comes with the packages pre-installed, and is still pretty much the same list.

1 Like

Sorry, I don’t understand what your point is.

Of course now that the package ecosystem has more or less caught up, using 1.0.2 (and soon 1.1) is the recommended solution. But at the time of the 1.0 release, while a lot of packages were still in the process of being upgraded, continuing on 0.6.x, with or without JuliaPro, was a perfectly viable solution. Nothing was taken away, and everyone who was more sensitive to the disruption of upgrading could wait it out.

Post 1.0, the release process is now more formalized (see especially the “Risk Tolerance Personas” part).

The one criticism I still find valid is the PkgEval infrastructure is still not up to public API / user consumption. It is used internally, but the official website is still the old one.

3 Likes

My point(s) are the following: new users to Julia should not be wise to start with v0.6.x, given the number of changes that happened between it and v1.0

As I said to a previous respondent, I have no truck with the progress made on upgrading packages to v1.0, I had always realised that this is not a trivial task, which was why v0.7 was muted as a medium term solution.

My article was about the information not being provided in allowing new users to find, select and use the Julia eco-system.

As a question, you say that a number of packages have been brought up to work with v1.0.x, which I’m sure is true, Can you give me some idea of the percentage and where I can get a definitive list?

It seems that you’re saying that it should be a bad idea to suggest v0.6, but given the ecosystem status that is what you’re inclined to do. Or am I misunderstanding you? What are the packages that makes you come to this judgement? It’s been a long time since key packages were not ready. You will probably find that some packages that worked on v0.6 will never get updated because the developers moved on somehow. That’ll always be true though. Some people are developing v1.0 packages now that might never work on v2.0 because they leave the ecosystem before that. Most packages are MIT licensed though, so it’s quite easy to fork and continue the developing if the package is important to someone.

Like the other people, it’s not clear to me what made you change your sentiment towards Julia. I was surprised by the julia v1.0 release as well, but I think it’s been for the better when we take everything into consideration.

1 Like

I am not sure who suggested starting with 0.6, or how this came up.

I don’t think this question is well-defined, unless you clarify the universe of packages. Eg as it happens frequently with free software, some packages are apparently abandoned, some since 0.4-0.5, and (I guess) are unlikely to be updated to 1.0, unless someone steps in to do it.

All the packages I use on regular basis have been updated, most within a month of 1.0. Many have seen multiple (minor) releases since. I think that at this point it is safe to say that if a package has not been updated to 1.0, the chances of it happening are very slim. So in a certain sense, nearly 100% of the actively maintained packages are updated (by construction :wink:).

This probably has to do with the fact that Julia is a relatively young language, and for some tasks solutions are WIP. That said, users who ask here or on other forums with a specific problem in mind usually get advice and suggestions.

I’ve not changed my sentiment to Julia, only a reluctance at present to talk about it to non-Julians.

The comment about v0.6 being dead was made to me, not by me, although I understand it; any companies using Julia in their Enterprise work will probably be best suited to shy away from v1.0 for the present time.

However given the number of changes catalogued in the Release Notes (and elsewhere) for v0.7/v1.0, I would not want to encourage new users to learn to write code radically different from that which would run on v1.0

An overall percentage isn’t meaningful because there is a long tail of packages which were used one time by one user (the package author), and those are the packages which generally get less maintenance. However, if you take a look at the 200 or 300 packages which have generally gotten the most use in the Julia v0.6 and v1.0 timeframe, I would be surprised if there are any that aren’t updated now. At least, I haven’t noticed any in a long time. So at this point, among the packages that most users actually install, close to 100% of packages work on v1.0. I say “close” because there might be one or two that aren’t, but I can’t name any package off the top of my head that hasn’t upgraded now that JuliaDB is done. Can you?