The State of the Julia Ecosystem

Huh? Didn’t I say the opposite of that?

Also I just checked and Winston.jl works on v1.x, so I’m not sure what the problem is there.

1 Like

I only mentioned Winston in request for an example and as I’m giving some attention to roughing out my chapter on visualization and was finding that on some of the computers it worked and others it did not.

I too do not know what the problem is but as they are OSX, I assume it may be a problem with Homebrew installations but also mentioned that I would probably remove/replace the old sections.

I have not tried with Windows or Linux

Fair enough, I have now removed references to it from the Julia home page:

which was only one aspect of my posting - of which I took a great deal of care and feel is entirely fair

I’m still unclear on what the other points were, aside from being unhappy about the 1.0 release process and the global scope change. What else should we do differently going forward?

Julia Computing is an enterprise which should be able to provide the resources to evaluation the status of its packages.

We will certainly take it into consideration. This may not be universally clear—although we certainly try to make it so—but Julia Computing is a for-profit commercial entity which is separate from the Julia project and is in no way obliged to run, manage or provide resources to the Julia project or ecosystem. To the extent that the company does so, it is because it seems like good business. I do think that it could make good business sense to address some of these issues and provide services to the open source Julia community for free, but the resulting services will quite possibly not be open source.

4 Likes

Stefan

It is late over here and a Friday, so I’ll look at your comments in the morning.

Perhaps I’ll just pop down to Downing Street and advise Mrs May on leaving the European Union.

2 Likes

Thanks Stefan, useful script…!

7 Likes

Some stats (because why not?):

  • 83% (2000/2381) of packages that have ever been registered work on 0.6
  • 62% (1483/2381) of packages that have ever been registered work on 1.x
  • 59% (1177/2000) of packages that worked on 0.6 also work on 1.x

Note that these are the claimed compatibilities—they may not be fully accurate. So 0.6.3 is still the best release if your metric is “if I pick a package at random, will it work?” However, 1.x releases are not far behind, and since people don’t pick packages are random, if you need a package, there’s a very good chance that it works on 1.x.

What’s interesting and a little unintuitive is that if you are migrating from 0.6 to 1.0 then things look a bit less rosy: the chance of picking a random package that worked on 0.6 and having it work on 1.x is only 59%. This happens because there are many packages that only work on 1.x and never worked on 0.6. I expect that some packages that worked on 0.6 will never be updated for 1.0 because they are abandoned.

(There are a small number of packages that worked on 0.1-0.5 but not on 0.6 or 1.0—only 75, however, which is about 3% of all packages ever registered.)

3 Likes

I think one of the things that gets missed in these discussions is that when the language was very, very young (which was not very long ago) everybody just made every package they felt like purely for the fun of it. As a little bit of time passed, the community coalesced around some combination of the most performant, most actively maintained and easiest to contribute to packages. Lots of people independently made the decisions like “I wrote a package for solving a very particular type of differential equation, and that was all good and fun, but DifferentialEquations.jl is relatively mature and a standard in the communtiy, so let me just fix whatever issues I might have had with that and abandon my package because it was fun when I wrote it but not very useful in the long run.” I have made analogous choices a few times myself. This results in a lot of abandoned packages that possess functionality that nevertheless exists in the current ecosystem.

So, I think if it were possible to make a plot of the amount of “functionality” that’s available in Julia it would be monotonically increasing with version number.

21 Likes

There are 1,492 packages that are registered and installable in Julia 1.0 according to a script I had written. It does differ from @StefanKarpinski’s by 10, but I think the number is about it (I ran it a few days ago). The website does state about 1,900 Julia packages, but it includes legacy packages that are not available in the current stable version. Something more important than the number of packages is the functionality. For example, can you convert PDF to text files in Julia, and you make HTTP requests, is there a tabular representation, and you do X. Having those functionalities documented and those packages discovered is quite useful and important. Maybe at the registry level or curated lists or some other tool.

3 Likes

Cxx.jl now works on both Linux and macOS(RFC: Fix source build(macOS), REPL mode and misc. upgrading oversights by Gnimuc · Pull Request #400 · JuliaInterop/Cxx.jl · GitHub). In fact, you don’t even need to build the dependency locally, see Make Cxx.jl ready for production environments post Julia 1.0.0 · Issue #390 · JuliaInterop/Cxx.jl · GitHub

BTW, I also tried cross-compiling a Windows version, but no luck.

3 Likes

Thanks, I’ll amend my Chapter 5 accordingly

It covers all the usual suspects, C, Python, R, Java + interfacing via the O/S and will be good to showcase C++ which is hard.

I won’t test on Windows but will mention it if still the same when (if?) I even get to press.

I’m still unclear on what the other points were, aside from being unhappy about the 1.0 release process and the global scope change. What else should we do differently going forward?

Well that makes three.

As for the launch of v1.0, excepting my own selection of adjective, I merely quoted comments about it immediately after JuliaCon, yourself included.

I think I’ve said I’d like to see more attention to the information supplied, particularly when it is misleading rather than just non-existent - which is why I selected the title of the article as I did. I made a number of other points in my presentation to LJuUG, which I did not include for the purposes of brevity.

but Julia Computing is a for-profit commercial entity which is separate from the Julia project and is in no way obliged to run, manage or provide resources to the Julia project or ecosystem

I appreciate that Julia Computing is a commercial enterprise, but its success will be reflected in the uptake of Julia. So the support that you give to the community as a whole surely will pay dividends in the long run. I guess there is an interesting parallel between yourselves and Redhat: RHEL vs the Centos and Fedora distributions.

I have no idea what article you’re referring to or what its title might be. Are you talking about the title of this thread, “The State of The Julia Ecosystem”? If so then that suggests a rather broad topic which as far as I can determine you are generally unhappy about? I’ve noticed that threads with such grand and existentially dramatic titles rarely end up being productive.

Well that makes three.

I only listed two things… one of which nothing can be done about without a time machine. Is the primary actionable point here that you are unhappy about the global scope change and would like us to do something about it? I am genuinely trying to understand.

It’s helpful if complaints or criticisms are specific and remediable. For example,

Complaint: pkg.julialang.org is out of date.
Short term fix: remove home page links to it (done).
Longer term fix:

  • replace it with something better that is maintained by an entity such as Julia Computing which can commit to long term support (in progress).

Complaint: I don’t like the new default global scope behavior.
Short term fix: SoftGlobalScope.jl (done).
Longer term fix:

  • discuss potential fixes (done),
  • decide on a plan (done), and
  • implement it to see how it works (in progress).

Are there other concrete actionable issues?

10 Likes

For a proper definition of ‘works’. For a package that is linked by the julialang.org i’d expect a release and passing tests.

Yes, I have not like GSD from the first time I came upon it for reasons which has been well chronicled. I think I did say that I applauded the fact that there was some movement on this, so complaint is your noun not mine.

I also said that I did not expect any general approbation, but my comments are made with the best of intentions.

I don’t think your intentions are the focus of the discussion, rather the lack of a clear and actionable summary of the issues. Julia is a collaborative effort; voicing dissatisfaction about various issues is most helpful only if it is done in a clear, concise, and actionable manner, as outlined above. Also, it is usually better to be concrete than general, because it is more likely to result in a solution.

Also, please kindly note that not all participants on this forum are native speakers of English. I am sure it is fun to talk like a Regency novel, but I wonder if this is the right place for it.

2 Likes

I find it hard to apologise for the way that I write English - is that not Regency enough for you?

2 Likes

I think you misunderstand — I did not ask you to apologize, and of course you write the way you want to.

It is just that keeping it simpler would be more inclusive.

The way I write is a function of the generation into which I was born and the education I received, it is not intended ever to appear trite

Since I believe (passionately) about the development of Julia, I am sorry to hear that you feel my comments are in any way exclusive, since they will clearly not be well appreciated by the audience for which they are directed, whether they approve of them or no.

Let’s please set aside language. That’s quite the distraction. I know I like my sloppy colloquial ‘mericanisms. The only one that I can’t grok is the brexit reference — is this really a comparable situation?


Back to the topic at hand:

Rather selectively, and missing some key points of context IMO. The biggest context, of course, is that they were all made three-four months ago and the situation is now much different. I personally don’t find those quotes very relevant anymore — except to help us inform how to improve making another breaking release (which we won’t be doing in the near future).

I think you also misattributed some of the quotes; for example search for “what’s done is done” brings up this comment from Tim Holy: List of working packages on v. 1.0 - #51 by tim.holy

I encourage re-reading Tim’s entire post there. There definitely were trade-offs in how we released 0.7 and 1.0. There weee some obvious pain points, but there were some big advantages, too. Not only did packages migrate quite quickly, but releasing them simultaneously also meant that the two were versions as nearly identical as we could make them — and that was an essential property to ensure upgrading through 0.7 worked as seamlessly as possible.

At this point I feel like we’re largely on the other side of the rocky transition — where I find I’m feeling the benefits from the double release much more than I experience the detractions.

And so I’m left puzzling over the purpose here. What outcomes would you consider a success as a result of this post?

8 Likes

Perhaps not.
Mrs May has been having trouble with her views here in Parliament and after returning from Europe yesterday.

It was just on the telly as I was typing.

Remember it’s Pantotime here in the UK, is that another essentially British institution, as well as our political parties imploding? A bit obscure I must admit.