The State of the Julia Ecosystem

As keno mentioned in this comment:

Still to be figured out is a new and improved build process using BinaryBuilder. Afterwards I’ll tag a new version.

I don’t think Cxx.jl will get tagged any sooner since Windows support is tough.

All tests passed locally on Julia-1.1. As for 1.0, there are 3 failure cases due to move `Future.copy!` to Base · Issue #29173 · JuliaLang/julia · GitHub

I think the underlying issue here is mismatched expectations. Without concrete, accurate expectations set and broadcast by responsible parties, someone’s best guess becomes community wisdom.


I wrote a longer post but decided against posting it as I wasn’t sure how harsh it would sound.

2 Likes

I don’t consider my expectations to be someone else’s responsibility. They are my own.

Also, I think that information can be disseminated, but expectations are formed by people themselves.

In the free software world, all tools come with the implicitly understood caveat that there is no guaranteed support, development, or bugfixes. If I depend on a particular piece of free software, I should be ready to step in and fix bugs, forking the project if necessary, pay for someone else to do this, or be ready to continue without it.

If I care about Julia’s development and release process, I should be ready to participate in discussions and make well-reasoned arguments (and PRs, which are the best way to make a point), with the understanding that the consensus of core developers may be different from what I would have preferred. If I want a set of curated packages, I should start an effort or contribute to an existing one.

7 Likes

It seems to me that the expectation mismatch is coming from people who don’t really get how the package ecosystem works. For packages, there is no centrally organized responsibility here. The Julia language itself has some central organization and responsibility, but the overall package ecosystem does not.

The problem is likely that this is not properly conveyed to new users, especially new users who might not be familiar with how free software development works. The Julia website could provide an introduction to this decentralized model of software development, since the effort of package maintainers is not part of one single centralized responsible team, but rather depends on the skill / time / funding of individuals / companies that are for a large part entirely separate / independent from the central responsibility of the Julia language.

1 Like

If you would not be at hominem, trying to exhaust oponent with “clever” questions or attacking his/her language, it could be useful catharsis. (I don’t say it could not bring pain)

I am curious what is your position! :slight_smile:


Reading this discourse we could find ideas useful for thinking about Julia ecosystem.

Evolution on isolated island and brexit have something in common with making language independent from other languages (aka 2nd language “problem”).

If you are not big mass land empire which could force your laws to others (and it is not current position) - means you are not able to make competitive ecosystem then you could end up with trying to glue/use/(make agreement with) 3rd and 4th language to get workable solution.

It is clear that some naive expectation will make some adopters/voters/evaluators frustrated. Others will be satisfied (some no matter what) and some could wait forever for promised bright future.

"You will know them by their fruits. Grapes are not gathered from thorn bushes nor figs from thistles, are they? "So every good tree bears good fruit, but the bad tree bears bad fruit. Matthew 7:15-20

This topic is about current state (of theJulia Ecosystem). If somebody is expecting good ecosystem answer that nobody is responsible for it doesn’t solve the problem of missing fruits.

Are we able to hear non popular honest words? Malcolm is saying that he feel current situation as less optimistic than 5 years ago.

Is it really only his problem?

Do you really think people are not aware about different models in free software development?

What precisely do you wont to introduce? (Something like that everybody who wants to use Linux has to be able to make PR for kernel like @Tamas_Papp is trying to convince here :scream: )

What do you mean by “central responsibility of the Julia language” precisely? What does that mean in distributed free software development for you?

Idon’t think you understood what I wrote.

My point was that if I want to use Linux, I either recognize that

  1. I depend on the kindness of strangers for bug fixes and new features, and consequently have no claim on their time,
  2. do the work myself, or
  3. pay someone to do it.

If I choose the first option, I am not in the position to demand or expect anything. Any attempts to this effect will come across as ridiculous and are likely to be ignored (probably much more explicitly on the kernel mailing list than here). I can disguise my attempts as appeals to some notion of responsibility towards users, or act as a stakeholder “concerned about the future of the language”, but I would not expect either to have a strong effect.

10 Likes

People who come from a proprietary software background, such as matlab users, might not be aware of how free software works. So yes, some people might not be aware. I have encountered many people, including professor’s who I work with, whom don’t understand how free software is developed, because they are not programmers. I had to explain to them how free software works.

I don’t want to introduce anything here, you seem to misunderstand me. Why would I think that everyone who uses Linux kernel should be able to make a PR to it? I didn’t imply anything like that. What I was talking about is the knowledge about what you can expect from a free software package ecosystem.

Each repository has its own structure for governance. Most of the Julia language packages were developed independently from the main Julia language effort. This means that the package ecosystem development is not centralized. The developers of the Julia language don’t have direct control over all package authors. This is not some sort of opinion or viewpoint, this is just a statement of a fact.

You can’t expect a developer who is not paid, someone who does work for free and shares it widely, to be responsible for the desires of random strangers on the internet. Also the developers who are paid to make free software aren’t responsible for fulfilling all the wishes of random strangers on the internet, because the free software license does not guarantee that responsibility.

Of course, it would be nice to coordinate the development and actions, but the responsibility is not centralized, it is decentralized and distributed among separate repositories and projects.

7 Likes

Interestingly, I think a lot of this discussion also comes down to differences in expectations. I definitely do not expect all the users thinking of switching are aware of the things @Tamas_Papp mentioned. Even if you’re coming from a well established free software ecosystem like R or Python, it’s possible to spend your entire career as a scientist that does some programming without ever encountering or considering these issues.

That said, if someone does understand, having it explained can seem like condescension. We should all read each other’s words in the spirit of charity.

This whole thread is people hearing and taking it seriously. Disagreeing with his outlook is not the same as dismissing his feelings. It can seem that way when people are digging into the reasons for those feelings, but they’re doing that to try to understand and fix. That can be really frustrating when it’s not about the nail.

1 Like

Who is expecting this???

I am afraid that some people don’t see difference between:

  1. analyze/opinion about current situation
    and
  2. blaming people for not doing what they are obliged to do

Could you see difference between these two?

Is it possible to say here something not optimistic without being accused for

  • beeing rude to people working for free
  • not understanding how free SW development work
  • not making PRs
  • using complicated language
  • to be not constructive enough

?

I like this:

it is same for anybody here. We are not obliged to do that.

Malcolm wrote book. He is doing revision for Julia 1.0 and what he met during this work does not strengthen his optimism about Julia. Is he free to say it? (If not him than who?)

He is not obliged to take responsibility for Julia ecosystem, not obliged to make packages or PRs to make things better, not obliged to do work on proposing concrete actions, does he?

2 Likes

There is no censorship here, as far as I am aware. Personally, I am not criticizing anyone for stating their perception of the situation here. I’m just providing food for thought and did not accuse anyone of anything.

1 Like

:smiley: :smiley: video is old but still good ( but she is Monica not Julia! :stuck_out_tongue_winking_eye: )

And thank you for these worlds! :slight_smile:

What I see in the OP (original post) is not only an opinion which could be quoted with

what I met during my work does not strengthen my optimism about Julia because reasons 1,2,3

but I see a lot of hyperbole, opinions stated as facts and belittleing speach without need. Example quotes:

This has come to be termed the “Julia Global Debacle” , while it remains in place the damage persists.

current state of the so-called Ecosystem

after the ill-judged launch of version 1.0 in August

but I suppose because the proponents are running out of stamina

and so it’s going on and on.
Thats why I responded immediately in the opposite tone (at least I tried).

To say it clearly, for me the OP is a rant and does not contain much information.

It is impressing how friendly all replies are and it is clear that there is REAL interest and curiosity out there why the OP (original poster) has his feelings, but I can’t see any enlightenment from the OP. This frustrates me but who am I to make this an issue.

Is he free to say it?

Sure he is. q.e.d. this thread.
But how he did is not the way to start a good discussion with the aim to make things better, but still everyone is free to do it as they wish. And even more the response is friendly and sympathetic.

What else could give more hope for the future of the julia world?

16 Likes

I am sorry to write once more to my original post (OP)

Possibly all OP’s run out of steam when they deteriorate into a critique of an author’s literary style or his/her movitation.

I do however feel an accusation of hyperbole in my OP is a little unfair, so, at risk of repetition list my principal points:

  1. The GSD was a term not coined by me, although I was, and remain, a critic of its introduction for reasons previously given.

  2. The original purpose of v0.7 was stated as giving developers time to bring their packages up to compliance by including deprecation warnings, which v1.0 would not and does not do, however it exited DEV less than 2 days prior to the launch of the latter

  3. The information on the compliance of packages was, and remains, incorrect.

… are these then exaggerations?

If the manner in which they were expressed masked my opinions, that was clearly not my intention.

7 Likes

Thank you for writing once again; please don’t be sorry! These further followups are quite helpful.

I really hope you don’t feel overly badgered here — most of us are honestly trying to get to the bottom of your critiques and trying to figure out how to best address them.

7 Likes

Although I like spirit of your message (and I am glad you show me your motivation behind your first reply here) I have to say that I am not sure that all replies were friendly.

I don’t want to dig into this deeper. Why I think some replies were (we could hope that unintentional) toxic could be understood from what I already wrote.

1 Like

Hello,

please feel free to move this to its own thread.

As a user and in the light of some posts in this thread I always found the relationship between “julia the MIT licensed language” (julialang.org ?) and the for-profit juliacomputing confusing, especially considering that some (?) core developers are involved in juliacomputing.

Eventually it might be a good thing, especially considering that Stefan Karpinski starts his reply with “This may not be universally clear …”, if there would be some more upfront expectation management concerning the role of juliacomputing and the vision for the interplay between “julia the language”, the julia ecosystem and juliacomputing from the juliacomputing perspective ?

Please note: I have no problem with juliacomputing being for-profit at all, especially as a user who is happy that I can use julia for free. Of course I have to admit on the other hand that I have an implied hope that juliacomputing will improve the julia language made available for free by one mechanism or another. Which brings the point back to expectation management.

This of course also carries a potential for confusion (especially after RH was bought by IBM) and I can understand why you draw a parallel here. However, this “triangle” has been established so long ago that implied expectations are mostly evident and upfront for the users of the distributions involved. With RHEL you get (hopefully) the level of support you pay for, and Fedora and CentOS simply is GNU/GPL “without any warranty …” which I hope most people understand by now.

Finally, please excuse any probably bad wording in advance, I am not a native speaker and I am reaching my limits here.

Best Regards

5 Likes

I don’t know people’s backgrounds, but a lot of the issues discussed in length in this thread are very common in any open-source community. I understand a lot of people really only download and use whatever software they need, without dealing with the list-serve, Stackexchange, or any other line of communication available for a particular project. There are also many mature projects that don’t have an active community anymore. In my personal case, I encountered pretty quickly in R that RStudio and R (the language) where separated entities, something that I know isn’t clear to many users of RStudio. Obviously RStudio has a lot riding in the success and spread of R and they are helping the language as a whole, but I’m not sure how much is RStudio driving the development of R. Same with other projects, like Revolution R and things like that. I assume the same will happen between Julia and Julia Computing. Anyway, my point is that sometimes I find users that have the same questions and doubts I had when I started getting into open-source projects.

Once you leave the confines of Windows and SPSS (in my case), you really enter a jungle of ideas, voices and ways of doing things that can be crazy. And there’s just no way of getting everybody into the same page, and even communities with “benevolent dictators” have issues. And we deal with them, like we’re doing here, writing posts, commenting, criticizing, making PR’s, whatever. This is the beauty of open-source, you can make it your own. Obviously if you know what you’re doing and you’re good at programming, you have a huge advantage over people that can use an Excel spreadsheet ant that’s it, but resources are there to get over the hump.

Anyway, I just think these threads really turn into exercises of circular arguments. I don’t believe opinions have changed radically in the weeks this thread has been open. And, as a regular user of the forum, I’ve learned to recognize some people’s positions regarding subjects like the state of the ecosystem, the “global scope debacle”, etc. Hope everybody remain cordial, this community if growing everyday and we will continue getting people unhappy for whatever reason and we will keep getting comments that are akin to the Yelp review that says: “I wish I could give this restaurant 0 stars! The restaurant was beautiful, the food was delicious, but the waiter took 4 minutes to bring water when I asked him to and that ruined the whole experience!”. And you just have to deal with that and keep improving.

And thanks everybody that develops Julia and its packages, you’re all doing a tremendous job.

14 Likes

Why not writing it like this at first? It is much clearer like that. And it is much less a wall of text and people will actually read it…

… and yes, there is some truth in what you write. I confess that my first reply (second post) does contain some toxic side effects which have been by purpose. I am sorry for that, it is sometimes better to just stay silent. On the other hand, I would say that the original post does trigger this.

1 Like